# Calico Cloud - Full Documentation > Complete documentation for Calico Cloud (version 22-2). --- ## About Calico ### About Calico ## What is Calico?[​](#what-is-calico) Calico is a single platform for networking, network security, and observability for any Kubernetes distribution in the cloud, on-premises, or at the edge. Whether you're just starting with Kubernetes or operating at scale, Calico's open source, enterprise, and cloud editions provide the networking, security, and observability you need. The key advantages of Calico are: - A single platform to address all networking, network security, and observability needs for Kubernetes environments - Consistent networking and network security controls for any Kubernetes distribution, ensuring workload portability - Ability to scale networking and network security to multi-cluster applications, VMs, and bare metal servers without additional software The Calico portfolio of products includes Calico Open Source, Calico Enterprise (self-managed), and Calico Cloud (fully-managed SaaS). Calico Cloud Free Tier is a free version of Calico Cloud that focuses on observability and policy management for a single cluster. All of this is built on Calico Open Source, the most widely used container networking and security solution. ## Calico overview[​](#calico-overview) ### Networking[​](#networking) ##### [Ingress Gateway](https://docs.tigera.io/calico-cloud/networking/ingress-gateway/about-calico-ingress-gateway) [Enterprise-grade traffic control based on K8s Gateway API](https://docs.tigera.io/calico-cloud/networking/ingress-gateway/about-calico-ingress-gateway) ##### [Egress Gateway](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-on-prem) [Secure outbound traffic with fixed, routable IP assignment](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-on-prem) ##### [Cluster Mesh](https://docs.tigera.io/calico-cloud/multicluster/overview) [Resilient and secure networking between clusters at scale](https://docs.tigera.io/calico-cloud/multicluster/overview) ### Network security[​](#network-security) ##### [DNS Policies](https://docs.tigera.io/calico-cloud/network-policy/domain-based-policy) [Simplify policy creation using DNS namess](https://docs.tigera.io/calico-cloud/network-policy/domain-based-policy) ##### [Staged Policies](https://docs.tigera.io/calico-cloud/network-policy/staged-network-policies) [Preview and test policies prior to deployment](https://docs.tigera.io/calico-cloud/network-policy/staged-network-policies) ##### [Policy Tiers](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/tiered-policy) [Enforce consistent, network policies with defined precedence](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/tiered-policy) ##### [Encryption](https://docs.tigera.io/calico-cloud/compliance/encrypt-cluster-pod-traffic) [High-performance encryption with WireGuard for data in transit](https://docs.tigera.io/calico-cloud/compliance/encrypt-cluster-pod-traffic) ### Threat detection, observability, and incident response[​](#threat-detection-observability-and-incident-response) ##### [Web Application Firewall](https://docs.tigera.io/calico-cloud/threat/web-application-firewall) [Detects malicious traffic targeting web applications, defending against exploits like SQL injection and cross-site scripting.](https://docs.tigera.io/calico-cloud/threat/web-application-firewall) ##### [Visualize traffic](https://docs.tigera.io/calico-cloud/observability/visualize-traffic) [Monitor, visualize, log, and quickly troubleshoot Kubernetes traffic](https://docs.tigera.io/calico-cloud/observability/visualize-traffic) ##### [Packet Capture](https://docs.tigera.io/calico-cloud/observability/packetcapture) [Self-service packet capture for troubleshooting and forensics](https://docs.tigera.io/calico-cloud/observability/packetcapture) ##### [Alerts & Incident Response](https://docs.tigera.io/calico-cloud/observability/alerts) [Alert on security events and deploy mitigating policies](https://docs.tigera.io/calico-cloud/observability/alerts) ### CI/CD automation and tools[​](#cicd-automation-and-tools) ##### [Policy Board](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/tour) [Author, view and manage Kubernetes network policies](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/tour) ##### [Policy recommendations](https://docs.tigera.io/calico-cloud/network-policy/recommendations/learn-about-policy-recommendations) [Automatically generate policies to isolate namespaces](https://docs.tigera.io/calico-cloud/network-policy/recommendations/learn-about-policy-recommendations) ##### [Multi-Cluster Controls](https://docs.tigera.io/calico-enterprise/latest/multicluster/set-up-multi-cluster-management/standard-install/create-a-management-cluster) [Manage network security and observability for multiple clusters](https://docs.tigera.io/calico-enterprise/latest/multicluster/set-up-multi-cluster-management/standard-install/create-a-management-cluster) [Enterprise](https://docs.tigera.io/calico-enterprise/latest/multicluster/set-up-multi-cluster-management/standard-install/create-a-management-cluster) ## Calico product editions[​](#calico-product-editions) ##### Calico Open Source Open-source networking and security for containers and Kubernetes ##### Calico Cloud Free Tier Observability & policy management for a single cluster ##### Calico Cloud SaaS platform for Kubernetes networking and security ##### Calico Enterprise Self-managed platform for Kubernetes networking and security ## Which product edition is right for me?[​](#which-product-edition-is-right-for-me) | My needs | Calico product edition | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | I want open source, best-in-class networking, network security, and observability capabilities that can work across any Kubernetes distribution, for free. | Calico Open Source [Get Started](https://docs.tigera.io/calico/latest/getting-started) | | I’m a Calico Open Source user who wants to leverage some of the improved observability and policy management capabilities that are available in Calico Cloud, for free. | Calico Cloud Free Tier [Sign up](https://auth.calicocloud.io/u/signup/identifier?state=hKFo2SBoMWV0M1RBUDlvWU83V2pGOS1ybVpCdmJGcDJRXzRSdaFur3VuaXZlcnNhbC1sb2dpbqN0aWTZIEJMclZBR2NmZzFacHRMclRIOFBzVEdNdVozOFlOSkpio2NpZNkgcTZ4MVM2dHdxZ0pHOVZLSEQzRGR0S0tEQTB2MlNuYkU) | | My organization wants a **fully managed SaaS** platform for network security and observability. | Calico Cloud [Get Started](https://docs.tigera.io/calico-cloud/get-started/) | | My organization wants a **self-managed platform** for network security and observability. | Calico Enterprise [Get Started](https://docs.tigera.io/calico-enterprise/latest/getting-started/) | ## Feature comparison matrix[​](#feature-comparison-matrix) | | Calico Open Source | Calico Cloud Free Tier\* | Calico Cloud | Calico Enterprise | | --------------------------------------------------------------------------- | ------------------ | ------------------------ | ----------------- | ----------------- | | **Management and Support** | | | | | | Mutli-cluster security controls management | | | | | | Data retention | In-memory | 24 hours | 7 days | Unlimited | | Number of clusters | Unlimited | One | Unlimited | Unlimited | | Number of users | N/A | One | Unlimited | Unlimited | | Support and maintenance | Community-driven | Community-driven | Standard/Business | Standard/Business | | **Networking** | | | | | | High performance, scalable pod networking | | | | | | Advanced IP address management | | | | | | Direct infrastructure peering without the overlay | | | | | | eBPF data plane | | | | | | Windows data plane | | | | | | nftables data plane | | | | | | iptables data plane | | | | | | VPP data plane | | | | | | Multiple Calico networks on a pod | | | | | | Dual ToR peering | | | | | | Ingress gateway | | | | | | Egress gateway | | | | | | Cluster mesh | | | | | | **Network Security** | | | | | | Seamless support for Kubernetes network policy | | | | | | Label-based policies for K8s and non-K8s workloads | | | | | | Namespace and cluster-wide scope | | | | | | Global default deny policy design | | | | | | Application layer policy | | | | | | Policy for services | | | | | | Policy board | | View only | | | | DNS/FQDN-based policy | | | | | | Hierarchical tiered network policy | | | | | | Policy recommendations | | Manual workflow | | | | Staged network policy | | | | | | Preview staged policies | | | | | | Network sets to limit IP ranges for egress and ingress traffic to workloads | | | | | | Data-in-transit encryption | | | | | | Universal firewall integration | | | | | | Workload-based IDS/IPS | | | | | | Deep packet inspection | | | | | | DDoS protection | | | | | | Workload-centric WAF | | | | | | Compliance reporting and alerts | | | | | | SIEM integrations | | | | | | **Network Security for VMs and Bare Metal** | | | | | | Restrict traffic to/from hosts and VMs using network policy | | | | | | Automatic host endpoints | | | | | | Apply policy to host-forwarded traffic | | | | | | **Observability** | | | | | | Flow logs API | | | | | | Calico Whisker web console | | | | | | Dynamic service and threat graph | | | | | | Application level observability | | | | | | Dynamic packet capture | | | | | | Flow visualizer | | | | | | Logs (flow) | | | | | | Logs (http traffic, audit, bgp, dns, events) | | | | | | Dashboards | | \*\* | | | | Alerts | | | | | \* Calico Cloud Free Tier requires a cluster with Calico Open Source 3.30 or higher. \*\* Calico Cloud Free Tier includes some of the dashboards that are available in Calico Cloud and Calico Enterprise. ## How to get started with Calico[​](#how-to-get-started-with-calico) Calico powers 100M+ containers across 8M+ nodes in 166 countries, and is supported across all major cloud providers and Kubernetes distributions. ### Ready to get started?[​](#ready-to-get-started) Start a free trial or request a demo to see Calico in action. - [Sign up for a Calico Cloud Free Tier account](https://www.calicocloud.io/) - [Schedule a 30-minute demo with our experts](https://www.tigera.io/demo/) ### Installation guides[​](#installation-guides) - [Install Calico Open Source](https://docs.tigera.io/calico/latest/getting-started) - [Install Calico Cloud Free Tier](https://docs.tigera.io/calico-cloud/free/connect-cluster-free) - [Install Calico Cloud](https://docs.tigera.io/calico-cloud/get-started/install-cluster) - [Install Calico Enterprise](https://docs.tigera.io/calico-enterprise/latest/getting-started/) ## How to engage[​](#how-to-engage) ### Learning resources[​](#learning-resources) - [Blog](https://www.tigera.io/blog/) - [Certifications](https://www.tigera.io/lp/calico-certification/) (self-paced) - [Product tutorials](https://www.tigera.io/tutorials/) (self-paced) - [Learn guides](https://www.tigera.io/learn/guides/kubernetes-networking/) - [Webinars and workshops](https://www.tigera.io/events/) (live and on demand) - [Resources](https://www.tigera.io/resources/) ### Get involved[​](#get-involved) - [Calico Open Source Community](https://www.tigera.io/project-calico/community/) - GitHub - [Project Calico](https://github.com/projectcalico/calico) - [Tigera](https://github.com/tigera) ### Get in touch[​](#get-in-touch) - [Slack](https://calicousers.slack.com/) (Calico Open Source users) - [YouTube](https://www.youtube.com/channel/UCFpTnXDNcBoXI4gqCDmegFA) (@ProjectCalico) - LinkedIn - [Tigera](https://www.linkedin.com/company/tigera/) - [Project Calico](https://www.linkedin.com/company/project-calico/) - X - [Tigera](https://x.com/tigeraio) - [Project Calico](https://x.com/projectcalico) - [Contact us](https://www.tigera.io/contact/) ### Calico product editions ##### Calico Open Source Open-source networking and security for containers and Kubernetes ##### Calico Cloud Free Tier Observability & policy management for a single cluster ##### Calico Cloud SaaS platform for Kubernetes networking and security ##### Calico Enterprise Self-managed platform for Kubernetes networking and security ## Which product edition is right for me?[​](#which-product-edition-is-right-for-me) | My needs | Calico product edition | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | I want open source, best-in-class networking, network security, and observability capabilities that can work across any Kubernetes distribution, for free. | Calico Open Source [Get Started](https://docs.tigera.io/calico/latest/getting-started) | | I’m a Calico Open Source user who wants to leverage some of the improved observability and policy management capabilities that are available in Calico Cloud, for free. | Calico Cloud Free Tier [Sign up](https://auth.calicocloud.io/u/signup/identifier?state=hKFo2SBoMWV0M1RBUDlvWU83V2pGOS1ybVpCdmJGcDJRXzRSdaFur3VuaXZlcnNhbC1sb2dpbqN0aWTZIEJMclZBR2NmZzFacHRMclRIOFBzVEdNdVozOFlOSkpio2NpZNkgcTZ4MVM2dHdxZ0pHOVZLSEQzRGR0S0tEQTB2MlNuYkU) | | My organization wants a **fully managed SaaS** platform for network security and observability. | Calico Cloud [Get Started](https://docs.tigera.io/calico-cloud/get-started/) | | My organization wants a **self-managed platform** for network security and observability. | Calico Enterprise [Get Started](https://docs.tigera.io/calico-enterprise/latest/getting-started/) | ## Feature comparison matrix[​](#feature-comparison-matrix) | | Calico Open Source | Calico Cloud Free Tier\* | Calico Cloud | Calico Enterprise | | --------------------------------------------------------------------------- | ------------------ | ------------------------ | ----------------- | ----------------- | | **Management and Support** | | | | | | Mutli-cluster security controls management | | | | | | Data retention | In-memory | 24 hours | 7 days | Unlimited | | Number of clusters | Unlimited | One | Unlimited | Unlimited | | Number of users | N/A | One | Unlimited | Unlimited | | Support and maintenance | Community-driven | Community-driven | Standard/Business | Standard/Business | | **Networking** | | | | | | High performance, scalable pod networking | | | | | | Advanced IP address management | | | | | | Direct infrastructure peering without the overlay | | | | | | eBPF data plane | | | | | | Windows data plane | | | | | | nftables data plane | | | | | | iptables data plane | | | | | | VPP data plane | | | | | | Multiple Calico networks on a pod | | | | | | Dual ToR peering | | | | | | Ingress gateway | | | | | | Egress gateway | | | | | | Cluster mesh | | | | | | **Network Security** | | | | | | Seamless support for Kubernetes network policy | | | | | | Label-based policies for K8s and non-K8s workloads | | | | | | Namespace and cluster-wide scope | | | | | | Global default deny policy design | | | | | | Application layer policy | | | | | | Policy for services | | | | | | Policy board | | View only | | | | DNS/FQDN-based policy | | | | | | Hierarchical tiered network policy | | | | | | Policy recommendations | | Manual workflow | | | | Staged network policy | | | | | | Preview staged policies | | | | | | Network sets to limit IP ranges for egress and ingress traffic to workloads | | | | | | Data-in-transit encryption | | | | | | Universal firewall integration | | | | | | Workload-based IDS/IPS | | | | | | Deep packet inspection | | | | | | DDoS protection | | | | | | Workload-centric WAF | | | | | | Compliance reporting and alerts | | | | | | SIEM integrations | | | | | | **Network Security for VMs and Bare Metal** | | | | | | Restrict traffic to/from hosts and VMs using network policy | | | | | | Automatic host endpoints | | | | | | Apply policy to host-forwarded traffic | | | | | | **Observability** | | | | | | Flow logs API | | | | | | Calico Whisker web console | | | | | | Dynamic service and threat graph | | | | | | Application level observability | | | | | | Dynamic packet capture | | | | | | Flow visualizer | | | | | | Logs (flow) | | | | | | Logs (http traffic, audit, bgp, dns, events) | | | | | | Dashboards | | \*\* | | | | Alerts | | | | | \* Calico Cloud Free Tier requires a cluster with Calico Open Source 3.30 or higher. \*\* Calico Cloud Free Tier includes some of the dashboards that are available in Calico Cloud and Calico Enterprise. ## How to get started with Calico[​](#how-to-get-started-with-calico) Calico powers 100M+ containers across 8M+ nodes in 166 countries, and is supported across all major cloud providers and Kubernetes distributions. ### Ready to get started?[​](#ready-to-get-started) Start a free trial or request a demo to see Calico in action. - [Sign up for a Calico Cloud Free Tier account](https://www.calicocloud.io/) - [Schedule a 30-minute demo with our experts](https://www.tigera.io/demo/) ### Installation guides[​](#installation-guides) - [Install Calico Open Source](https://docs.tigera.io/calico/latest/getting-started) - [Install Calico Cloud Free Tier](https://docs.tigera.io/calico-cloud/free/connect-cluster-free) - [Install Calico Cloud](https://docs.tigera.io/calico-cloud/get-started/install-cluster) - [Install Calico Enterprise](https://docs.tigera.io/calico-enterprise/latest/getting-started/) ## How to engage[​](#how-to-engage) ### Learning resources[​](#learning-resources) - [Blog](https://www.tigera.io/blog/) - [Certifications](https://www.tigera.io/lp/calico-certification/) (self-paced) - [Product tutorials](https://www.tigera.io/tutorials/) (self-paced) - [Learn guides](https://www.tigera.io/learn/guides/kubernetes-networking/) - [Webinars and workshops](https://www.tigera.io/events/) (live and on demand) - [Resources](https://www.tigera.io/resources/) ### Get involved[​](#get-involved) - [Calico Open Source Community](https://www.tigera.io/project-calico/community/) - GitHub - [Project Calico](https://github.com/projectcalico/calico) - [Tigera](https://github.com/tigera) ### Get in touch[​](#get-in-touch) - [Slack](https://calicousers.slack.com/) (Calico Open Source users) - [YouTube](https://www.youtube.com/channel/UCFpTnXDNcBoXI4gqCDmegFA) (@ProjectCalico) - LinkedIn - [Tigera](https://www.linkedin.com/company/tigera/) - [Project Calico](https://www.linkedin.com/company/project-calico/) - X - [Tigera](https://x.com/tigeraio) - [Project Calico](https://x.com/projectcalico) - [Contact us](https://www.tigera.io/contact/) --- ## Calico Cloud Free Tier ### Calico Cloud Free Tier ##### [Overview](https://docs.tigera.io/calico-cloud/free/overview) [Overview of Calico Cloud Free Tier](https://docs.tigera.io/calico-cloud/free/overview) ##### [Quickstart](https://docs.tigera.io/calico-cloud/free/quickstart) [Quickstart guide for Calico Cloud Free Tier.](https://docs.tigera.io/calico-cloud/free/quickstart) ##### [Connect a cluster](https://docs.tigera.io/calico-cloud/free/connect-cluster-free) [Securely connect your cluster to Calico Cloud Free Tier to access centralized network observability for your Kubernetes deployment.](https://docs.tigera.io/calico-cloud/free/connect-cluster-free) ##### [Remove a cluster](https://docs.tigera.io/calico-cloud/free/disconnect-cluster-free) [Disconnect and remove your cluster from Calico Cloud Free Tier](https://docs.tigera.io/calico-cloud/free/disconnect-cluster-free) ### Calico Cloud Free Tier Learn about the observability tools available to you when you connect your Calico Open Source cluster to the Calico Cloud Free Tier web console. ## About Calico Cloud Free Tier[​](#about-calico-cloud-free-tier) Calico Cloud Free Tier provides essential observability tools for understanding and securing your Kubernetes cluster. With these tools, you can monitor network traffic, visualize service interactions, and gain insights into the behavior of your network policies. A free account with Calico Cloud allows you to connect one cluster to the free-tier web console. The cluster must be running Calico Open Source 3.30 or later. Features available in the free-tier web console are similar to those in the pro-tier console, but with some limitations. For example, you can view your network policies, but you cannot edit them directly in the console, as you can with Calico Cloud Pro. As a general rule, the free-tier console allows you to view what's going on in your cluster. To make changes, such as creating or editing policies, you must use `kubectl` commands in your cluster. ### What happens when you connect your cluster?[​](#what-happens-when-you-connect-your-cluster) Calico Open Source 3.30 introduced a new component, Goldmane, which exposes a gRPC API with aggregated flow logs from your cluster. This is the data source that feeds [Calico Whisker](https://docs.tigera.io/calico/latest/observability/view-flow-logs), an in-cluster web console that displays a stream of flows. Calico Cloud Free Tier uses the same flow logs API to receive data from your cluster. When you connect your cluster to Calico Cloud Free Tier, you apply manifests that establish a secure connection to the Calico Cloud management cluster. These manifests create a `ManagementClusterConnection` resource in your cluster, which specifies the address of the Calico Cloud management cluster, and two `Secret` resources that provide certificates for secure communication. Unlike Whisker, which can only display a stream of flows in real time, Calico Cloud Free Tier collects and retains flow logs for a 24-hour period. You can sign in to the console at any time to view data from the last 24 hours. If you choose to disconnect and remove your cluster from the console, your data is deleted after a day. And because you're not installing anything complicated in your cluster, you can easily remove the resources and restore you cluster to its previous state. ## Feature overview[​](#feature-overview) Calico Cloud Free Tier provides a set of observability features that help you monitor and manage your cluster's network traffic. ### Dashboards[​](#dashboards) Dashboards help you see and understand the activity in your cluster using categorized graphs and charts. They visually represent data from your logs, providing a birds-eye view of cluster health, traffic volume, DNS activity, and performance metrics. This allows for quick assessment of the overall state of your cluster's network. ![The traffic volume dashboard.](https://docs.tigera.io/img/calico-cloud/free-overview-1.png) *Figure 1: The traffic volume dashboard* ### Service Graph[​](#service-graph) Service Graph provides a point-to-point, topographical representation of network traffic within your cluster. It serves as the primary tool for visualizing how services communicate and for troubleshooting connectivity issues. You can click on nodes or connections to see detailed information and associated logs for targeted analysis. ![The Service Graph page.](https://docs.tigera.io/img/calico-cloud/free-overview-2.png) *Figure 2: The Service Graph page.* ### Policies[​](#policies) Network policy is the primary tool for securing a Kubernetes network by controlling traffic flow. Policies restrict network traffic (egress and ingress) in your cluster so only the traffic that you explicitly allow can flow. Calico Cloud supports standard Kubernetes policies and its own enhanced network policies for fine-grained control. ![The Policies page.](https://docs.tigera.io/img/calico-cloud/free-overview-3.png) *Figure 3: The Policies page.* For more information, see the Calico Open Source [network policy documentation](https://docs.tigera.io/calico/latest/network-policy/). ### Policy recommendations[​](#policy-recommendations) Policy recommendations help you secure your Kubernetes environments by automatically suggesting network policies based on observed network traffic. This proactively identifies necessary communication flows and enables you to implement a least-privilege security model with minimal effort. The recommendations are generated by analyzing flow logs. You can activate the recommendations as staged network policies for testing before you use them in production. ![The Policy Recommendations page.](https://docs.tigera.io/img/calico-cloud/free-overview-3-5.png) *Figure 4: The Policy Recommendations page.* ### Network sets[​](#network-sets) Network sets are resources for defining IP subnetworks or CIDRs, which can then be easily referenced in network policies using standard label selectors. They are a powerful feature for simplifying policy management, especially when dealing with external network ranges or grouping common IP addresses. Using network sets makes policies more reusable and scalable. For more information, see: - [Understanding network sets](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/networksets). - Calico Open Source resource documentation on [Network set](https://docs.tigera.io/calico/latest/reference/resources/networkset) and [Global network set](https://docs.tigera.io/calico/latest/reference/resources/globalnetworkset). ## Next steps[​](#next-steps) - [Sign up for a free account with Calico Cloud](https://calicocloud.io/). - [Tutorial: Quickstart guide to Calico Cloud Free Tier](https://docs.tigera.io/calico-cloud/free/quickstart). - [Connect a cluster to Calico Cloud Free Tier](https://docs.tigera.io/calico-cloud/free/connect-cluster-free) - For an overview of the expanded feature set in Calico Cloud Pro, take a [tour of the full web console](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/tour). ### Calico Cloud Free Tier quickstart guide This quickstart guide shows you how to connect a Kubernetes cluster to Calico Cloud Free Tier. You'll learn how to create a cluster with `kind`, connect that cluster to the Calico Cloud web console, and use observability tools to monitor network traffic. ## Before you begin[​](#before-you-begin) - You need to [sign up for a Calico Cloud Free Tier account](https://calicocloud.io). - You also need to install a few tools to complete this tutorial: - `kind`. This is what you'll use to create a cluster on your workstation. For installation instructions, see the [`kind` documentation](https://kind.sigs.k8s.io/docs/user/quick-start/). - Docker Engine or Docker Desktop. This is required to run containers for the `kind` utility. For installation instructions, see the [Docker documentation](https://docs.docker.com/desktop/). - `kubectl`. This is the tool you'll use to interact with your cluster. For installation instructions, see the [Kubernetes documentation](https://kubernetes.io/docs/tasks/tools/#kubectl) ## Step 1: Create a cluster[​](#step-1-create-a-cluster) In this step, you will: - **Create a cluster:** Use `kind` to create a Kubernetes cluster. - **Verify the cluster:** Check that the cluster is running and ready. 1. Create a file called `config.yaml` and give it the following content: ```bash kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 nodes: - role: control-plane - role: worker - role: worker networking: disableDefaultCNI: true podSubnet: 192.168.0.0/16 ``` This configuration file tells `kind` to create a cluster with one control-plane node and two worker nodes. It instructs `kind` to create the cluster without a CNI. The `podSubnet` range defines the IP addresses that Kubernetes will use for pods. 2. Start your Kubernetes cluster with the configuration file by running the following command: ```bash kind create cluster --name=calico-cluster --config=config.yaml ``` `kind` reads you configuration file and creates a cluster in a few minutes. Expected output ```bash Creating cluster "calico-cluster" ... ✓ Ensuring node image (kindest/node:v1.29.2) 🖼 ✓ Preparing nodes 📦 📦 📦 ✓ Writing configuration 📜 ✓ Starting control-plane 🕹️ ✓ Installing StorageClass 💾 ✓ Joining worker nodes 🚜 Set kubectl context to "kind-calico-cluster" You can now use your cluster with: kubectl cluster-info --context kind-calico-cluster Thanks for using kind! 😊 ``` 3. To verify that your cluster is working, run the following command: ```bash kubectl get nodes ``` You should see three nodes with the name you gave the cluster. Expected output ```bash NAME STATUS ROLES AGE VERSION calico-cluster-control-plane NotReady control-plane 5m46s v1.29.2 calico-cluster-worker NotReady 5m23s v1.29.2 calico-cluster-worker2 NotReady 5m22s v1.29.2 ``` Don't wait for the nodes to get a `Ready` status. They remain in a `NotReady` status until you configure networking in the next step. ## Step 2. Install Calico[​](#step-2-install-calico) In this step, you will install Calico in your cluster. 1. Install the Tigera Operator and custom resource definitions. ```bash kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.31.2/manifests/tigera-operator.yaml ``` Expected output ```bash namespace/tigera-operator created serviceaccount/tigera-operator created clusterrole.rbac.authorization.k8s.io/tigera-operator-secrets created clusterrole.rbac.authorization.k8s.io/tigera-operator created clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created rolebinding.rbac.authorization.k8s.io/tigera-operator-secrets created deployment.apps/tigera-operator created ``` 2. Install Calico Cloud by creating the necessary custom resources. ```bash kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.31.2/manifests/custom-resources.yaml ``` Expected output ```bash installation.operator.tigera.io/default created apiserver.operator.tigera.io/default created goldmane.operator.tigera.io/default created whisker.operator.tigera.io/default created ``` 3. Monitor the deployment by running the following command: ```bash watch kubectl get tigerastatus ``` After a few minutes, all the Calico components display `True` in the `AVAILABLE` column. Expected output ```bash NAME AVAILABLE PROGRESSING DEGRADED SINCE apiserver True False False 4m9s calico True False False 3m29s goldmane True False False 3m39s ippools True False False 6m4s whisker True False False 3m19s ``` ## Step 3. Connect to Calico Cloud Free Tier[​](#step-3-connect-to-calico-cloud-free-tier) In this step, you will connect your cluster to Calico Cloud Free Tier. 1. Sign in to the Calico Cloud web console and click the **Connect a cluster** button on the welcome screen. 2. Follow the prompts to create a name for your cluster (for example, `quickstart-cluster`) and copy a `kubectl` command to run in your cluster. What's happening in this command? This command creates three resources in your cluster: - **A `ManagementClusterConnection` resource**. This resource specifies the address of the Calico Cloud management cluster. - **A `Secret` resource (`tigera-managed-cluster-connection`)**. This resource provides certificates for secure communication between your cluster and the Calico Cloud management cluster. - **A `Secret` resource (`tigera-voltron-linseed-certs-public`)**. This resource provides certificates for secure communications for the specific components that Calico Cloud uses for log data and observability. Example of generated kubectl command to connect a cluster to Calico Cloud Free Tier ```bash kubectl apply -f - <1: A screenshot of the Managed Clusters page showing the new cluster connection.* 5. On the left side of the console, click **Service Graph** to view the Service Graph, which we'll be using to view your network traffic in this tutorial. What you see is a dynamic diagram of the namespaces in your cluster and the connections between them. For now, it shows only system namespaces. ![A screenshot of the Service Graph showing system namespaces.](https://docs.tigera.io/img/calico-cloud/quickstart-2.png) *Figure 2: A screenshot of the Service Graph showing system namespaces.* We'll return to this page to see the traffic after we deploy a sample application in the next step. ## Step 4. Deploy NGINX and BusyBox to generate traffic[​](#step-4-deploy-nginx-and-busybox-to-generate-traffic) Now it's time to generate some network traffic. We'll do this first by deploying an NGINX server and exposing it as a service in the cluster. Then we'll make HTTP requests from another pod in the cluster to the NGINX server and to an external website. For this we'll use the BusyBox utility. In this step, you will: - **Create a server:** Deploy an NGINX web server in your Kubernetes cluster. - **Expose the server:** Make the NGINX server accessible within the cluster. - **Test connectivity:** Use a BusyBox pod to verify connections to the NGINX server and the public internet. 1. Create a namespace for your application: ```bash kubectl create namespace quickstart ``` Expected output ```bash namespace/quickstart created ``` 2. Deploy an NGINX web server in the `quickstart` namespace: ```bash kubectl create deployment --namespace=quickstart nginx --image=nginx ``` Expected output ```bash deployment.apps/nginx created ``` 3. Expose the NGINX deployment to make it accessible within the cluster: ```bash kubectl expose --namespace=quickstart deployment nginx --port=80 ``` Expected output ```bash service/nginx exposed ``` 4. Start a BusyBox session to test whether you can access the NGINX server. ```bash kubectl run --namespace=quickstart access --rm -ti --image busybox /bin/sh ``` This command creates a BusyBox pod inside the `quickstart` namespace and starts a shell session inside the pod. Expected output ```bash If you don't see a command prompt, try pressing enter. / # ``` 5. In the BusyBox shell, run the following command to test communication with the NGINX server: ```bash wget -qO- http://nginx ``` You should see the HTML content of the NGINX welcome page. Expected output ```html Welcome to nginx!

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

``` This confirms that the BusyBox pod can access the NGINX server. 6. In the Busybox shell, run the following command test communication with the public internet: ```bash wget -qO- https://docs.tigera.io/pod-connection-test.txt ``` You should see the content of the file `pod-connectivity-test.txt`. Expected output ```html You successfully connected to https://docs.tigera.io/pod-connection-test.txt. ``` This confirms that the BusyBox pod can access the public internet. 7. In the web console, go to the **Service Graph** page to view your flow logs. It may take up to 5 minutes for the flows to appear. When they appear, click the `quickstart` namespace in the Service Graph to filter the view to show the flows only in that namespace. In the list of flows, you should see three new connection types: one to `coredns` one to `nginx`, and another to `pub`, meaning "public network". ![Service Graph with \`quickstart\` namepace selected showing flows to NGINX and public network](https://docs.tigera.io/img/calico-cloud/quickstart-3.png) *Figure 3: Service Graph with `quickstart` namepace selected showing flows to NGINX and public network.* ## Step 5. Restrict all traffic with a default deny policy[​](#step-5-restrict-all-traffic-with-a-default-deny-policy) To effectively secure your cluster, it's best to start by denying all traffic, and then gradually allowing only the necessary traffic. We'll do this by applying a Global Calico Network Policy that denies all ingress and egress traffic by default. In this step, you will: - **Implement a global default deny policy:** Use a Global Calico Network Policy to deny all ingress and egress traffic by default. - **Verify access is denied:** Use your BusyBox pod to confirm that the policy is working as expected. 1. Create a Global Calico Network Policy to deny all traffic except for the necessary system namespaces: ```bash kubectl create -f - <4: Service Graph showing denied flows to `coredns`.* By following these steps, you have successfully implemented a global default deny policy and verified that it is working as expected. ## Step 6. Create targeted network policy for allowed traffic[​](#step-6-create-targeted-network-policy-for-allowed-traffic) Now that you have a default deny policy in place, you need to create specific policies to allow only the necessary traffic for your applications to function. The `default-deny` policy blocks all ingress and egress traffic for pods not in system namespaces, including our `access` (BusyBox) and `nginx` pods in the `quickstart` namespace. In this step, you will: - **Allow egress traffic from BusyBox** Create a network policy to allow egress traffic from the BusyBox pod to the public internet. - **Allow ingress traffic to NGINX** Create a network policy to allow ingress traffic to the NGINX server. 1. Create a Calico network policy in the `quickstart` namespace that selects the `access` pod and allows all egress traffic from it. ```bash kubectl create -f - < Welcome to nginx!

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

``` You have now successfully implemented a default deny policy while allowing only the necessary traffic for your applications to function. ## Step 7. Clean up[​](#step-7-clean-up) 1. To delete the cluster, run the following command: ```bash kind delete cluster --name=calico-cluster ``` Expected output ```bash Deleted cluster: calico-cluster ``` 2. To remove the cluster from Calico Cloud, go to the **Managed Clusters** page. Click **Actions > Disconnect**, and in the confirmation dialog, click **I ran the commands**. (Ordinarily you would run the commands from the dialog, but since you deleted the cluster already, you don't need to do this.) 3. Click **Actions > Remove** to fully remove the cluster from Calico Cloud. You can now connect another cluster to make use of the observability tools. ## Additional resources[​](#additional-resources) - To view requirements and connect another cluster, see [Connect a cluster to Calico Cloud Free Tier](https://docs.tigera.io/calico-cloud/free/connect-cluster-free). ### Connect a cluster to Calico Cloud Free Tier This page shows you how to connect your Kubernetes cluster to Calico Cloud Free Tier. > **SUCCESS:** Don't have a cluster ready to connect? Try the [Calico Cloud Free Tier quickstart guide](https://docs.tigera.io/calico-cloud/free/quickstart) to create a cluster locally and connect it to Calico Cloud Free Tier. ## Prerequisites[​](#prerequisites) - You have a Kubernetes cluster. - Your `kubectl` tool is configured to communicate with your cluster. - If you're working in a restricted network environment, you are able to create allow rules for TCP egress traffic to the Calico Cloud management plane. - You [installed Calico Open Source 3.30 or later on your cluster](https://docs.tigera.io/calico/latest/getting-started). - If you upgraded from an earlier version of Calico Open Source, you [enabled the `Goldmane` custom resource](https://docs.tigera.io/calico/latest/observability/enable-whisker). - Calico Open Source was installed using the operator method. Manifest- and Helm-based installations are not supported. Check Calico compatibility To confirm that you're on a compatible Calico setup, you can run the following command: ```bash kubectl get tigerastatus goldmane ``` If you're running Calico Open Source 3.30+ with the `Goldmane` custom resource is enabled, you should see something like this: Example output ```bash NAME AVAILABLE PROGRESSING DEGRADED SINCE goldmane True False False 21h ``` This cluster is ready to connect to Calico Cloud Free Tier. If instead you get an error (`Error from server (NotFound): tigerastatuses.operator.tigera.io "goldmane" not found`), then you need to do one of the following: - If you're already on 3.30+ [enable the `Goldmane` custom resource](https://docs.tigera.io/calico/latest/observability/enable-whisker). - If you're on 3.29 or earlier, [upgrade your Calico installation to 3.30 or later](https://docs.tigera.io/calico/latest/operations/upgrading/kubernetes-upgrade#upgrading-an-installation-that-uses-the-operator) and then [enable the `Goldmane` custom resource](https://docs.tigera.io/calico/latest/observability/enable-whisker). - You have an active Calico Cloud Free Tier account. To create an account, go to the [Calico Cloud Free Tier sign-up page](https://calicocloud.io). - You are signed in to the Calico Cloud web console. ## Connect your cluster[​](#connect-your-cluster) 1. From the web console, click the **Connect a cluster** button on the welcome screen. 2. Follow the prompts to create a name for your cluster and copy a `kubectl` command to run in your cluster. What's happening in this command? This command creates three resources in your cluster: - **A `ManagementClusterConnection` resource**. This resource specifies the address of the Calico Cloud management cluster. - **A `Secret` resource (`tigera-managed-cluster-connection`)**. This resource provides certificates for secure communication between your cluster and the Calico Cloud management cluster. - **A `Secret` resource (`tigera-voltron-linseed-certs-public`)**. This resource provides certificates for secure communications for the specific components that Calico Cloud uses for log data and observability. Example of generated kubectl command to connect a cluster to Calico Cloud Free Tier ```bash kubectl apply -f - < Delete**. 3. If you made changes to your network policies or firewall to allow egress traffic to Calico Cloud, you may want to revert those changes. ## Next steps[​](#next-steps) - [Connect a cluster to Calico Cloud Free Tier](https://docs.tigera.io/calico-cloud/free/connect-cluster-free) --- ## Install and upgrade ### Install and upgrade Requirements and guides for connecting your Kubernetes cluster to Calico Cloud. ## Before you begin[​](#before-you-begin) ##### [Calico Cloud architecture](https://docs.tigera.io/calico-cloud/get-started/cc-arch-diagram) [Understand the main components of Calico Cloud.](https://docs.tigera.io/calico-cloud/get-started/cc-arch-diagram) ##### [What happens when you connect a cluster to Calico Cloud](https://docs.tigera.io/calico-cloud/get-started/connect-cluster) [Get answers to your questions about connecting to Calico Cloud.](https://docs.tigera.io/calico-cloud/get-started/connect-cluster) ##### [System requirements](https://docs.tigera.io/calico-cloud/get-started/system-requirements) [Review cluster requirements to connect to Calico Cloud.](https://docs.tigera.io/calico-cloud/get-started/system-requirements) ##### [Prepare your cluster for Calico Cloud](https://docs.tigera.io/calico-cloud/get-started/prepare-cluster) [Prepare your cluster to install Calico Cloud.](https://docs.tigera.io/calico-cloud/get-started/prepare-cluster) ##### [Limitations and known issues for Windows nodes](https://docs.tigera.io/calico-cloud/get-started/windows-limitations) [Review limitations before starting installation.](https://docs.tigera.io/calico-cloud/get-started/windows-limitations) ## Connect your cluster[​](#connect-your-cluster) ##### [Install Calico Cloud](https://docs.tigera.io/calico-cloud/get-started/install-cluster) [Steps to connect your cluster to Calico Cloud.](https://docs.tigera.io/calico-cloud/get-started/install-cluster) ##### [Set up a private registry](https://docs.tigera.io/calico-cloud/get-started/setup-private-registry) [Add images to a private registry for installing Calico Cloud on a cluster.](https://docs.tigera.io/calico-cloud/get-started/setup-private-registry) ##### [Install using a private registry](https://docs.tigera.io/calico-cloud/get-started/install-private-registry) [Steps to connect your cluster to Calico Cloud.](https://docs.tigera.io/calico-cloud/get-started/install-private-registry) ##### [Install Calico Cloud as part of an automated workflow](https://docs.tigera.io/calico-cloud/get-started/install-automated) [Install Calico Cloud as part of an automated workflow.](https://docs.tigera.io/calico-cloud/get-started/install-automated) ##### [Prepare your cluster for Calico Cloud](https://docs.tigera.io/calico-cloud/get-started/prepare-cluster) [Prepare your cluster to install Calico Cloud.](https://docs.tigera.io/calico-cloud/get-started/prepare-cluster) ## Troubleshooting[​](#troubleshooting) ##### [Tigera Operator troubleshooting checklist](https://docs.tigera.io/calico-cloud/get-started/operator-checklist) [Additional troubleshooting for the Tigera Operator.](https://docs.tigera.io/calico-cloud/get-started/operator-checklist) ##### [Troubleshooting checklist](https://docs.tigera.io/calico-cloud/get-started/checklist) [Review this checklist before opening a Support ticket.](https://docs.tigera.io/calico-cloud/get-started/checklist) ## Upgrade[​](#upgrade) ##### [Upgrade Calico Cloud](https://docs.tigera.io/calico-cloud/get-started/upgrade-cluster) [Steps to upgrade to the latest version of Calico Cloud.](https://docs.tigera.io/calico-cloud/get-started/upgrade-cluster) ### Calico Cloud architecture ## Calico Cloud security[​](#calico-cloud-security) Calico Cloud architecture is based on the Calico Cloud multi-cluster management feature. Calico Cloud manages the control plane, and you connect your clusters (called **managed clusters**) to the control plane. Communication between the Calico Cloud control plane and managed clusters is secured using TLS tunnels. ![calico-architecture](https://docs.tigera.io/assets/images/cc-architecture-a7af17df4e191c766a2ea84aeedb3f55.svg) The components that secure communications between the Calico Cloud control plane and managed clusters are: - **Calico Cloud tunnel server** - accepts secure TLS connections from managed clusters - **Guardian** - an agent that runs in each managed cluster that proxies communication between Calico Cloud components and managed cluster components All connections go through the Calico Cloud tunnel server and Guardian. The only exception is during installation and upgrade when managed clusters connect to Calico Cloud using TLS connections to get install/update resources, register the cluster, and report the status of the install/update. The Calico Cloud tunnel is initiated by Guardian on the managed cluster. The Calico Cloud control plane does not initiate new connections to the managed cluster outside of the tunnel. However, there are connections that go through the Calico Cloud tunnel server that are initiated from the control plane; for example, when a user interacts with the web console, or when configuration needs to be pushed into the managed cluster. ## Managed cluster components[​](#managed-cluster-components) The following diagram shows the major components in a managed cluster, followed by component descriptions. ![calico-architecture-diagram](https://docs.tigera.io/assets/images/cc-arch-diagram-0acbffa24e27fda5fe2722b0ed1391bf.png) | Component | Description | Ports/Protocol | | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | Calico Cloud controller | Deploys required resources for Calico Cloud. | TCP 6443 to Kubernetes API server | | Calico Cloud installer | Gets installation resources from the Calico Cloud portal, registers a managed cluster, and reports installation or upgrade progress. | • TCP 443 to Calico Cloud hosted service • TCP 6443 to Kubernetes API server | | Calico Cloud tunnel server | Communicates with managed clusters by creating secure TLS tunnels. | Port 9000 from managed clusters | | calico-node | Bundles key components that are required for networking containers with Calico Cloud: • Felix • BIRD • confd | • TCP 5473 to Typha • TCP 9900 and 9081 from Prometheus API service | | Container threat detection | A threat detection engine that analyzes observed file and process activity to detect known malicious and suspicious activity. Monitors the following types of suspicious activity within containers: • Access to sensitive system files and directories • Defense evasion • Discovery • Execution • Persistence • Privilege escalation Includes these components: **Runtime Security Operator** An operator to manage and reconcile container threat defense components. **Runtime Reporter Pods** Pods running on each node in the cluster to perform the detection activity outlined above.They send activity reports to Elasticsearch for analysis by Calico Cloud. | TCP to Kubernetes API server | | Compliance | Generates compliance reports for the Kubernetes cluster. Reports are based on archived flow and audit logs for Calico Cloud resources, plus any audit logs you’ve configured for Kubernetes resources in the Kubernetes API server. Compliance reports provide the following high-level information: • Endpoints explicitly protected using ingress or egress policy • Policies and services - Policies and services associated with endpoints - Policy audit logs • Traffic - Allowed ingress/egress traffic to/from namespaces, and to/from the internet Compliance includes these components: **compliance-snapshotter** Handles listing of required Kubernetes and Calico Cloudconfiguration and pushes snapshots to Elasticsearch. Snapshots give you visibility into configuration changes, and how the cluster-wide configuration has evolved within a reporting interval. **compliance-reporter** Handles report generation. Reads configuration history from Elasticsearch and determines time evolution of cluster-wide configuration, including relationships between policies, endpoints, services, and network sets. Data is then passed through a zero-trust aggregator to determine the “worst-case outliers” in the reporting interval. **compliance-controller** Reads report configuration and manages creation, deletion, and monitoring of report generation jobs. **compliance-benchmarker** A daemonset that runs checks in the CIS Kubernetes Benchmark on each node so you can see if Kubernetes is securely deployed. | • TCP 8080 to Guardian • TCP 6443 to Kubernetes API server | | Fluentd | Open-source data collector for unified logging. Collects and forwards Calico Cloud logs (flows, DNS, L7) to log storage. | • TCP 8080 to Guardian • TCP 9080 from Prometheus API service | | Guardian | An agent running in each managed cluster that proxies communication between the Calico Cloud tunnel server and your managed cluster. Secured using TLS tunnels. | • Port 9000 to tunnel server • TCP 6443 to Kubernetes API server • TCP 6443 from Calico Cloud components | | Installation endpoints | Endpoints at `*.calicocloud.io` and `*.projectcalico.org`. | TCP 443 for both | | Intrusion detection controller | Handles integrations with threat intelligence feeds and Calico Cloud custom alerts. | • TCP 8080 to Guardian • TCP 6443 to Kubernetes API server | | Image Assurance | Identifies vulnerabilities in container images that you deploy to Kubernetes clusters. Components of interest are: **Admission controller** Uses Kubernetes Validating Webhook Configuration to control which images can be used to create pods based on scan results. **API** Isolates tenant data and authorizes all external access to Image Assurance data. **Note:** Calico Cloud does not store registry credentials in its database and does not pull customer images into the Calico Cloud control plane. | •  TCP 8080 to Guardian • TCP 6443 to Kubernetes API server | | Kubernetes API server | A Kubernetes component that validates and configures data for the API objects (for example, pods, services, and others). | TCP 6443 (from all components) | | kube-controllers | Monitors the Kubernetes API and performs actions based on cluster state. Calico Cloud kube-controllers container includes these controllers: • Node • Service • Federated services • Authorization | • TCP 9094 from Prometheus API service • TCP 6443 to Kubernetes API server | | Log storage | Storage for logs (flows, L7, DNS, audit). Data for each managed cluster is isolated and protected against unauthorized access. | n/a | | Packet capture API | Retrieves capture files (pcap format) generated by a packet capture for use with network protocol analysis tools like Wireshark. Packet capture data is visible in the web console and Service Graph. | • TCP 8449 Guardian to Packet Capture API • TCP 6443 to Kubernetes API server | | Prometheus API service | Collects metrics from Calico Cloud components and makes the metrics available to the web console. | • TCP 6443 to Kubernetes API server • TCP 9080 to Fluentd • TCP 9900 and 9081 to Prometheus API service | | Tigera API server | Allows users to manage Calico Cloud resources such as policies and tiers through kubectl or the Kubernetes API server. | • TCP 9095 to Prometheus API service • TCP 8080 from Kubernetes API server | | Typha | Increases scale by reducing each node’s impact on the datastore. | TCP 5473 from calico-node to Typha | | User access to the web console | Authenticated users can access the browser-based the web console, which provides network traffic visibility and troubleshooting, centralized multi-cluster management, threat-defense, container threat detection, policy lifecycle management, scan images for vulnerabilities, and compliance for multiple roles/stakeholders. | Port 443 to Calico Cloud tunnel server | ### What happens when you connect a cluster to Calico Cloud Although connecting your cluster to Calico Cloud is easy, we also understand you may want details about what happens when your cluster is managed by Calico Cloud. We hope this article, along with the other topics in this section, gives you the information you need to install Calico Cloud with confidence. If you have other questions, let us know: [Support policy](https://www.tigera.io/legal/calico-cloud-support-policy) or email: [feedback@calicocloud.io](mailto:feedback@calicocloud.io). ## What happens when you connect your cluster[​](#what-happens-when-you-connect-your-cluster) - Your cluster is registered and connected to Calico Cloud - Your Calico open source install is updated with resources for Calico Cloud features - Tigera components and services are added to collect metrics and logs that are sent to the Calico Cloud management plane, which provides the basis for visibility and troubleshooting - Policies are added to secure communications between Tigera components - A global threat feed is added to alert on any egress traffic to addresses in the threat feed to protect the cluster - All of your existing network policies (Kubernetes and Calico) can be found in a single place: the default tier After your cluster is connected, it is listed in the Managed Clusters page where you can move between multiple clusters. ![managed-clusters](https://docs.tigera.io/assets/images/managed-clusters-6de294f55b709c8f94c39c821beea02a.png) ## What happens when you disconnect your cluster[​](#what-happens-when-you-disconnect-your-cluster) Whether you’ve finished your Calico Cloud Trial, or terminated your licensed Calico Cloud subscription, we know you want your cluster to remain functional. Calico Cloud provides a migration script that returns your cluster to a working state in open-source Calico. The migration script: - Removes and cleans up all Calico Cloud components and services that have no equivalent in open-source Calico - Switches the operator configuration to open-source Calico, which migrates your cluster to a version of open-source Calico - Ensures all Calico policies are migrated to the default tier, or allows you to remove all Calico policies For details of the migration script, see [Uninstall Calico Cloud from a cluster](https://docs.tigera.io/calico-cloud/operations/disconnect). ## What happens under the covers when you connect your cluster[​](#what-happens-under-the-covers-when-you-connect-your-cluster) ### Pre-check[​](#pre-check) - Verifies that your cluster can be migrated, and validates that the cluster platform and version is supported by Calico Cloud - Records the number of nodes and other basic attributes of your cluster. This data remains on your system and is used for troubleshooting in case there are issues connecting your cluster. ### Install and connect[​](#install-and-connect) - Based on the Calico install for your cluster, the manifest is upgraded/migrated to use the Calico Cloud Tigera Operator. (Note that clusters installed using Helm are currently not supported.) - Tigera Operator installs the required Calico Cloud Custom Defined Resources (CRDs), and a standard pull secret to allow access to Calico Cloud images - Installs a Prometheus instance for component metrics (if it doesn’t already exist on the cluster) - Pushes a license to the cluster so components receive appropriate entitlements - Adds custom namespaces, service account, and role bindings to support cluster access and permissions to the Calico Cloud user interface - Creates RBAC permissions to access these resources: - Number of nodes in the cluster and stats for billing purposes - Policies in the cluster, and pod information - Registers the cluster so it can connect to Calico Cloud - Creates a global threat feed to alert on any egress traffic to addresses in the threat feed - Adds policies to secure communication for Calico Cloud components, including safeguards that prevent any new network policies from impacting the vital functions of the cluster - Creates roles and bindings to allow the installer to operate and to allow Calico Cloud to operate after installation. Each of these objects is given minimal permissions for its specific function. - Creates roles and bindings for two user types for access to the web console: - **Admin** - full permissions for network policies and Calico Cloud resources, and read permissions for namespace and pods - **User** - read-only/view permissions to same resources above ## How long does it take to connect a cluster?[​](#how-long-does-it-take-to-connect-a-cluster) Typically, about 5 minutes. ## Troubleshooting an installation[​](#troubleshooting-an-installation) - A checklist to [troubleshoot your installation](https://docs.tigera.io/calico-cloud/get-started/checklist) - Additional troubleshooting for the [Tigera Operator](https://docs.tigera.io/calico-cloud/get-started/operator-checklist) ### System requirements Before you connect your cluster to Calico Cloud, make sure your cluster meets the system requirements. Your cluster must already have a CNI installed before you can connect to Calico Cloud. ## Kubernetes distributions and CNIs[​](#kubernetes-distributions-and-cnis) Calico Cloud works with Kubernetes on self-provisioned infrastructure and on managed Kubernetes distributions. To use Calico Cloud for both networking and network policy, your cluster must have Calico Open Source installed before you connect to Calico Cloud. For most managed distributions, you can use the provider's CNI for networking and use Calico Cloud for network policy. | Distribution | Supported CNIs | | --------------------------------------------- | --------------------------------------------------- | | Kubernetes on self-provisioned infrastructure | - Calico Open Source 3.20 or later | | Amazon Elastic Kubernetes Service | - Calico Open Source 3.20 or later - Amazon VPC CNI | | Azure Kubernetes Service | - Calico Open Source 3.20 or later - Azure CNI | | Google Kubernetes Engine | - Calico Open Source 3.20 or later - GKE CNI | | Rancher Kubernetes Engine 2 | - Calico Open Source 3.20 or later | > **SECONDARY:** The Kubernetes distributions listed above are those that Tigera currently tests and supports for Calico Cloud. You may be able to connect clusters on other distributions with Calico Open Source installed as the CNI. For more information about connecting other cluster types to Calico Cloud, [contact Support](https://tigeraio.my.site.com/community/s/login/). ### Other distributions[​](#other-distributions) The Calico Cloud installer may be able to successfully connect ## Kubernetes versions[​](#kubernetes-versions) To install Calico Cloud v22.1.0, your Kubernetes distribution must be based on one of the following Kubernetes versions: - Kubernetes 1.34 - Kubernetes 1.33 - Kubernetes 1.32 - Kubernetes 1.31 Kubernetes compatibility for older versions of Calico Cloud | Calico Cloud version | Kubernetes versions | | -------------------- | ------------------- | | 22.1.0 | 1.31-1.34 | | 22.0.1 | 1.31-1.33 | | 22.0.0 | 1.31-1.33 | | 21.3.0 | 1.31-1.33 | | 21.2.0 | 1.30-1.31 | | 21.1.0 | 1.30-1.31 | | 21.0.0 | 1.30-1.31 | | 20.4.0 | 1.29-1.31 | | 20.3.1 | 1.29-1.30 | | 20.2.0 | 1.28-1.30 | | 20.1.0 | 1.28-1.30 | | 20.0.1 | 1.28-1.30 | | 20.0.0 | 1.28-1.30 | | 19.4.1 | 1.27-1.29 | | 19.4.0 | 1.27-1.29 | | 19.3.0 | 1.27-1.29 | | 19.2.0 | 1.26-1.29 | | 19.1.0 | 1.26-1.29 | | 19.0.0 | 1.26-1.28 | > **SUCCESS:** You can connect a cluster that uses a more recent (and unsupported) version of Kubernetes, but only for testing and development purposes. Connecting such a cluster is not suitable for production or any other workloads. To connect a cluster with an unsupported version of Kubernetes, start by following the standard installation procedure. When the installation fails, follow the instructions in the error message to connect your cluster. ## Architectures[​](#architectures) Calico Cloud can be installed on nodes based on the following chip architectures: - x86-64 - ARM64 ## Windows node support for hybrid clusters[​](#windows-node-support-for-hybrid-clusters) Calico Cloud supports hybrid Kubernetes clusters that use a Linux control plane with Windows nodes. You can install Calico Cloud on clusters of this type on the following managed distributions: - Amazon Elastic Kubernetes Service - Azure Kubernetes Service ## Browser support for the web console[​](#browser-support-for-the-web-console) To access the web console, you can use latest two versions of the following web browsers: - Chrome - Safari ## Kubernetes reconcilers[​](#kubernetes-reconcilers) - Calico Cloud cannot be usually be installed on clusters that are managed by any kind of Kubernetes reconciler (for example, Addon-manager). To verify, look for an annotation called `addonmanager.kubernetes.io/mode` on either of the following resources. (The resources may not exist). - `tigera-operator` deployment in the `tigera-operator` namespace - `calico-node` daemonset in the `kube-system` namespace If the following command finds addonmanager on either of the resources, then Addon-manager is being used. Find a different cluster to use. ```bash kubectl get -n -o yaml | grep ' addonmanager.kubernetes.io/mode:' ``` - Some AKS clusters with AddonManager are compatible with Calico Cloud. If output from the following command includes "EnsureExists", then the install is compatible with Calico Cloud. ```bash kubectl get CustomResourceDefinition installations.operator.tigera.io -o yaml | grep ' addonmanager.kubernetes.io/mode:' ``` > **SECONDARY:** If the command output does not include "EnsureExists" and you are on a recent version of AKS your cluster might still be compatible. You can [contact Support](https://tigeraio.my.site.com/community/s/login/) for more information. > **WARNING:** If your cluster already has Calico installed by AKS and managed by AddonManager, the standard [uninstall](https://docs.tigera.io/calico-cloud/operations/disconnect) is not supported. You will need to reach out to support to create a plan to uninstall Calico Cloud. ## Distribution-specific requirements[​](#distribution-specific-requirements) ### Azure Kubernetes Service[​](#azure-kubernetes-service) - Your cluster uses a supported combination of the `networkPlugin` and `networkPolicy` configurations: - `"networkPlugin": "none"` and `"networkPolicy": null` - `"networkPlugin": "azure"` and `"networkPolicy": null` - `"networkPlugin": "azure"` and `"networkPolicy": "calico"` You can check your configuration by running the following command: ```bash az aks show --query 'networkProfile' ``` - If your cluster uses the Azure CNI, your cluster's CNI is set to [transparent mode](https://docs.microsoft.com/en-us/azure/aks/faq#what-is-azure-cni-transparent-mode-vs-bridge-mode). ```bash az vmss run-command invoke -g -n --scripts "cat /etc/cni/net.d/*" --command-id RunShellScript --instance-id 0 --query 'value[0].message' ``` If the Azure CNI is enabled, the output should include `"mode": "transparent"`. ### Google Kubernetes Service[​](#google-kubernetes-service) - Your cluster's network policy is disabled. To verify, run the following command: ```bash gcloud container clusters describe --flatten addonsConfig.networkPolicyConfig.disabled ``` - Your cluster's Dataplane V2 is set to `null`. To verify, run the following command: ```bash gcloud container clusters describe --flatten networkConfig.datapathProvider ``` ### Rancher Kubernetes Engine 2[​](#rancher-kubernetes-engine-2) - The Calico Open Source CNI must not be provisioned by the RKE2 installer. You can connect an RKE2 cluster to Calico Cloud only if: - the RKE2 cluster was installed without a CNI - Calico Open Source was installed manually To verify, run this command to see the configuration on your control-plane node: ```bash cat /etc/rancher/rke2/config.yaml ``` You should see `cni: none`. If you're creating a new RKE2 cluster, you can set this configuration as an environment variable (`RKE2_CNI=none`) when you run the installation script. ## Next steps[​](#next-steps) - [Prepare your cluster for Calico Cloud](https://docs.tigera.io/calico-cloud/get-started/prepare-cluster) ### Prepare your cluster for Calico Cloud Get your cluster ready to connect to Calico Cloud. ## Prerequisites[​](#prerequisites) - Your cluster meets the [system requirements](https://docs.tigera.io/calico-cloud/get-started/system-requirements) for Calico Cloud. ## Allow outbound traffic from pods to Calico Cloud endpoints[​](#allow-outbound-traffic-from-pods-to-calico-cloud-endpoints) Pods running in your Kubernetes cluster must allow outbound traffic to the following endpoints: - `https://installer.calicocloud.io:443/*` - `https://www.calicocloud.io:443/api/*` - `https://client-auth.calicocloud.io:443/*` - TCP to `.calicocloud.io:9000` For each node, Docker must be able to pull images from the following endpoints: - `quay.io` - `cdn01.quay.io` - `cdn02.quay.io` - `us-docker.pkg.dev` ## Make sure you have the right permissions for your platform user account[​](#make-sure-you-have-the-right-permissions-for-your-platform-user-account) If your cluster is installed on a managed service, you must have sufficient permissions from your identity and access management system. Check that you are authorized to create the following Kubernetes resource types: - `ClusterRole` - `ClusterRoleBinding` - `Deployment` - `ServiceAccount` - `CustomResourceDefinition` ## Prepare your cluster on Azure Kubernetes Service[​](#prepare-your-cluster-on-azure-kubernetes-service) ### Remove taints from Linux node pools[​](#remove-taints-from-linux-node-pools) If you have a hybrid clusters with both Windows and Linux nodes, the Linux nodes may have taints that prevent Calico Cloud from scheduling pods on those nodes. These taints must be removed before you connect your cluster to Calico Cloud. You can check whether any node pools in your cluster have taints by running the following command: ```bash az aks nodepool list --resource-group --cluster-name --query "[].{name:name nodeTaints:nodeTaints}" ``` Remove any taints in the Linux node pools by running the command: ```bash az aks nodepool update --resource-group --cluster-name --name --node-taints "" ``` ## Prepare your cluster Google Kubernetes Engine[​](#prepare-your-cluster-google-kubernetes-engine) ### Turn on intranode visibility for your cluster[​](#turn-on-intranode-visibility-for-your-cluster) Verify that intranode visibility is set to `Enabled` by running the following command: ```bash gcloud container clusters describe --flatten networkConfig.enableIntraNodeVisibility ``` If intranode visibiliity is not enabled, you must enable it by following running the following command: ```bash gcloud container clusters update --enable-intra-node-visibility ``` ## Next steps[​](#next-steps) - [Connect your cluster to Calico Cloud](https://docs.tigera.io/calico-cloud/get-started/install-cluster) - [Connect your cluster using a private registry](https://docs.tigera.io/calico-cloud/get-started/install-private-registry) ### Connect a cluster to Calico Cloud You can quickly connect a cluster to Calico Cloud by generating a unique kubectl or Helm command in the web console and running it on your cluster. ## Prerequisites[​](#prerequisites) - You have an paid Calico Cloud account. - You are signed in to the web console as a user with the **Owner**, **Admin**, or **DevOps** role. - You have at least one cluster that meets our [system requirements](https://docs.tigera.io/calico-cloud/get-started/system-requirements). - You have kubectl access to the cluster. - If you're using Helm, you installed Helm 3.0 or later on your workstation. ## Connect a cluster to Calico Cloud with kubectl[​](#connect-a-cluster-to-calico-cloud-with-kubectl) 1. If your organization uses multiple projects to group managed clusters, click the **Project** menu and select the project you want your cluster to be part of. 2. From the **Managed Clusters** page, click **Connect Cluster**. 3. In the **Connect a Cluster** dialog, enter a **Cluster Name** and select a **Cluster Type**. 4. Optional: If you must install a specific older release, select the Calico Cloud version you want to install. We always recommend the latest version, which is installed by default. 5. Click **Connect** to generate a unique kubectl command. Copy the command. Use alternate manifest for legacy features The Image Assurance and Container Threat Detection features were removed for new users in Calico Cloud 21.1.0. Legacy users of those features can continue to use a deprecated version until the features are completely removed in a future release. To continue using these features, modify the generated command by replacing **two instances** of `deploy.yaml` with `deploy-with-container-security.yaml`. This change gives you a manifest with all three legacy features enabled. You cannot enable or disable these features individually. Example of generated kubectl command with alternate manifest ```bash kubectl apply -f https://installer.calicocloud.io/manifests/cc-operator/latest/deploy-with-container-security.yaml && curl -H "Authorization: Bearer ..." "https://www.calicocloud.io/api/managed-cluster/deploy-with-container-security.yaml?version=v22.1.0" | kubectl apply -f - ``` 6. From a terminal, paste and run the command. 7. On the **Managed Clusters** page, you should immediately see your cluster in the list of managed clusters. Monitor the status under **Connection Status**. When the status changes to **Connected**, installation is complete and your cluster is connected to Calico Cloud. ## Connect a cluster to Calico Cloud with Helm[​](#connect-a-cluster-to-calico-cloud-with-helm) 1. If your organization uses multiple projects to group managed clusters, click the **Project** menu and select the project you want your cluster to be part of. 2. From the **Managed Clusters** page, click **Connect Cluster**. 3. In the **Connect a Cluster** dialog, enter a **Cluster Name** and select a **Cluster Type**. 4. Optional: If you must install a specific older release, select the Calico Cloud version you want to install. We always recommend the latest version, which is installed by default. 5. Click **Connect** to generate a unique Helm installation command. Copy the command. Example of generated Helm installation command ```bash helm repo add calico-cloud https://installer.calicocloud.io/charts --force-update && helm upgrade --install calico-cloud-crds calico-cloud/calico-cloud-crds --namespace calico-cloud --create-namespace && helm upgrade --install calico-cloud calico-cloud/calico-cloud --namespace calico-cloud --set apiKey=ryl34elz8:9dav6eoag:ifk1uwruwlgp7vzn7ecijt5zjbf5p9p1il1ag8877ylwjo4muu19wzg2g8x5qa7x --set installer.clusterName=my-cluster --set installer.calicoCloudVersion=v19.1.0 ``` 6. Optional: If you want to enable the Packet Capture or Compliance Reports features, you can append `--set installer.components.packetCaptureAPI.state=Enabled` to the generated Helm command. You can change this option only by reinstalling or upgrading Calico Cloud and changing the values. | Feature | Key | Values | | ------------------ | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Packet Capture | `installer.components.packetCaptureAPI.state` | `Enabled`, `Disabled` (default\*) \* The default for new clusters is `Disabled`. For upgrades for previously connected clusters, the default will retain the previous state. | | Compliance Reports | `installer.components.compliance.enabled` | `true`, `false` (default) | Example of generated Helm command with user-added parameters ```bash helm repo add calico-cloud https://installer.calicocloud.io/charts --force-update && helm upgrade --install calico-cloud-crds calico-cloud/calico-cloud-crds --namespace calico-cloud --create-namespace && helm upgrade --install calico-cloud calico-cloud/calico-cloud --namespace calico-cloud --set apiKey=ryl34elz8:9dav6eoag:ifk1uwruwlgp7vzn7ecijt5zjbf5p9p1il1ag8877ylwjo4muu19wzg2g8x5qa7x --set installer.clusterName=my-cluster --set installer.calicoCloudVersion=v19.1.0 \ --set installer.components.packetCaptureAPI.state=Enabled ``` In this example, the command connects the cluster to Calico Cloud with the Packet Capture feature enabled. Use alternate feature keys for legacy features The Image Assurance and Container Threat Detection features were removed for new users in Calico Cloud 21.1.0. Legacy users of those features can continue to use a deprecated version until the features are completely removed in a future release. | Feature | Key | Values | | -------------------------- | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Image Assurance | `installer.components.imageAssurance.state` | `Enabled`, `Disabled` (default) | | Container Threat Detection | `installer.components.runtimeSecurity.state` | `Enabled`, `Disabled` (default\*) \* The default for new clusters is `Disabled`. For upgrades for previously connected clusters, the default will retain the previous state. | | Packet Capture | `installer.components.packetCaptureAPI.state` | `Enabled`, `Disabled` (default\*) \* The default for new clusters is `Disabled`. For upgrades for previously connected clusters, the default will retain the previous state. | | Compliance Reports | `installer.components.compliance.enabled` | `true`, `false` (default) | Example of generated Helm command with user-added parameters ```bash helm repo add calico-cloud https://installer.calicocloud.io/charts --force-update && helm upgrade --install calico-cloud-crds calico-cloud/calico-cloud-crds --namespace calico-cloud --create-namespace && helm upgrade --install calico-cloud calico-cloud/calico-cloud --namespace calico-cloud --set apiKey=ryl34elz8:9dav6eoag:ifk1uwruwlgp7vzn7ecijt5zjbf5p9p1il1ag8877ylwjo4muu19wzg2g8x5qa7x --set installer.clusterName=my-cluster --set installer.calicoCloudVersion=v19.1.0 \ --set installer.components.imageAssurance.state=Enabled \ --set installer.components.runtimeSecurity.state=Enabled \ ``` In this example, the command connects the cluster to Calico Cloud with Image Assurance and Container Threat Detection features enabled. 7. From a terminal, paste and run the command. 8. On the **Managed Clusters** page, you should immediately see your cluster in the list of managed clusters. Monitor the status under **Connection Status**. When the status changes to **Connected**, installation is complete and your cluster is connected to Calico Cloud. ## Additional resources[​](#additional-resources) - [Calico Cloud troubleshooting checklist](https://docs.tigera.io/calico-cloud/get-started/checklist) - [Tigera Operator troubleshooting checklist](https://docs.tigera.io/calico-cloud/get-started/operator-checklist) ### Connect a cluster to Calico Cloud using a private registry You can perform a Helm installation from images stored on a private registry. ## Prerequisites[​](#prerequisites) - You have an paid Calico Cloud account. - You are signed in to the web console as a user with the **Owner**, **Admin**, or **DevOps** role. - You have at least one cluster that meets our [system requirements](https://docs.tigera.io/calico-cloud/get-started/system-requirements). - You have kubectl access to the cluster. - You have installed Helm 3.0 or later on your workstation. - You have [added the Calico Cloud images to a private registry](https://docs.tigera.io/calico-cloud/get-started/setup-private-registry), and you have the following information about the registry: - Registry secret name > **SECONDARY:** If your private registry requires credentials, create a `calico-cloud` namespace on your cluster. Then, create an image pull secret and use this name for the **Registry Secret Name**. - Image registry - Image path ## Install Calico Cloud using a private registry[​](#install-calico-cloud-using-a-private-registry) 1. If your organization uses multiple projects to group managed clusters, click the **Project** menu and select the project you want your cluster to be part of. 2. From the **Managed Clusters** page, click **Connect Cluster**. 3. In the **Connect a Cluster** dialog, enter a **Cluster Name** and select a **Cluster Type**. 4. Optional: If you must install a specific older release, select the Calico Cloud version you want to install. We always recommend the latest version, which is installed by default. 5. Click **Advanced Options**, and then select both **Install via helm** and **Private registry**. 6. Enter the **Registry Secret Name**, **Image registry**, and **Image path**. 7. Click **Connect** to generate a unique Helm installation command. Copy the command. 8. Optional: If you want to enable the Packet Capture feature, you can append `--set installer.components.packetCaptureAPI.state=Enabled` to the generated Helm command. You can change this option only by reinstalling or upgrading Calico Cloud and changing the values. | Feature | Key | Values | | ------------------ | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Packet Capture | `installer.components.packetCaptureAPI.state` | `Enabled`, `Disabled` (default\*) \* The default for new clusters is `Disabled`. For upgrades for previously connected clusters, the default will retain the previous state. | | Compliance Reports | `installer.components.compliance.enabled` | `true`, `false` (default) | Example of generated Helm command with user-added parameters ```bash helm repo add calico-cloud https://installer.calicocloud.io/charts --force-update && helm upgrade --install calico-cloud-crds calico-cloud/calico-cloud-crds --namespace calico-cloud --create-namespace && helm upgrade --install calico-cloud calico-cloud/calico-cloud --namespace calico-cloud --set apiKey=ryl34elz8:9dav6eoag:ifk1uwruwlgp7vzn7ecijt5zjbf5p9p1il1ag8877ylwjo4muu19wzg2g8x5qa7x --set installer.clusterName=my-cluster --set installer.calicoCloudVersion=v19.1.0 \ --set installer.components.packetCaptureAPI.state=Enabled ``` In this example, the command connects the cluster to Calico Cloud with the Packet Capture feature enabled. Use alternate feature keys for legacy features The Image Assurance and Container Threat Detection features were removed for new users in Calico Cloud 21.1.0. Legacy users of those features can continue to use a deprecated version until the features are completely removed in a future release. | Feature | Key | Values | | -------------------------- | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Image Assurance | `installer.components.imageAssurance.state` | `Enabled`, `Disabled` (default) | | Container Threat Detection | `installer.components.runtimeSecurity.state` | `Enabled`, `Disabled` (default\*) \* The default for new clusters is `Disabled`. For upgrades for previously connected clusters, the default will retain the previous state. | | Packet Capture | `installer.components.packetCaptureAPI.state` | `Enabled`, `Disabled` (default\*) \* The default for new clusters is `Disabled`. For upgrades for previously connected clusters, the default will retain the previous state. | | Compliance Reports | `installer.components.compliance.enabled` | `true`, `false` (default) | Example of generated Helm command with user-added parameters ```bash helm repo add calico-cloud https://installer.calicocloud.io/charts --force-update && helm upgrade --install calico-cloud-crds calico-cloud/calico-cloud-crds --namespace calico-cloud --create-namespace && helm upgrade --install calico-cloud calico-cloud/calico-cloud --namespace calico-cloud --set apiKey=ryl34elz8:9dav6eoag:ifk1uwruwlgp7vzn7ecijt5zjbf5p9p1il1ag8877ylwjo4muu19wzg2g8x5qa7x --set installer.clusterName=my-cluster --set installer.calicoCloudVersion=v19.1.0 \ --set installer.components.imageAssurance.state=Enabled \ --set installer.components.runtimeSecurity.state=Enabled \ ``` In this example, the command connects the cluster to Calico Cloud with Image Assurance and Container Threat Detection features enabled. 9. From a terminal, paste and run the command. 10. On the **Managed Clusters** page, you should immediately see your cluster in the list of managed clusters. Monitor the status under **Connection Status**. When the status changes to **Connected**, installation is complete and your cluster is connected to Calico Cloud. ## Additional resources[​](#additional-resources) - [Calico Cloud troubleshooting checklist](https://docs.tigera.io/calico-cloud/get-started/checklist) - [Tigera Operator troubleshooting checklist](https://docs.tigera.io/calico-cloud/get-started/operator-checklist) ### Install Calico Cloud as part of an automated workflow You can connect clusters to Calico Cloud as part of an automated workflow, using persistent client credentials and customized Helm charts. ## Prerequisites[​](#prerequisites) - You have an paid Calico Cloud account. - You are signed in to the web console as a user with the **Owner** or **Admin**, role. - You have at least one cluster that meets our [system requirements](https://docs.tigera.io/calico-cloud/get-started/system-requirements). - You have kubectl access to the cluster. - You have installed Helm 3.0 or later on your workstation. ## Create client credentials[​](#create-client-credentials) Create client credentials and generate a Kubernetes secret to use for automated Helm installations. 1. Select the user icon **> Settings**. 2. Under the **Client Credentials** tab, click **Add Client Credential** 3. In the **Add Client Credential** dialog, enter a name and click **Create**. Your new client credential will appear in the list on the **Manage Client Credentials** page. 4. Locate the newly created client credential in the list and select **Action** > **Manage keys** > **Add Key** 5. Enter a name, choose how long the key will be valid, and click **Create key**. 6. Click **Download** to download the `.yaml` secret file and store it in a secure location. You will not be able to retrieve this secret again. > **INFO:** To ensure that you always have a valid key, you should transition to a second key before the first key expires. Create a second key, download the secret, and then replace copies of the secret file for the first key with the secret file for the second key. When all the secrets from the first key have been replaced, you can safely delete the first key from the **Client Credentials** page. When the key is deleted, all API requests based on that key will be rejected. ## About customizing your Helm installation[​](#about-customizing-your-helm-installation) You can customize your Calico Cloud installation for the following purposes: - to enable or disable certain features - to modify pod scheduling and resource management To do this, you can either edit the default `values.yaml` file or pass individual key-value pairs using the `--set` flag for the `helm upgrade` command. ### Required parameters[​](#required-parameters) The following paramaters are required for all Calico Cloud installations. | Parameter | Value | Example | Description | | ------------------------------ | ------ | -------------- | ------------------------------------------------------- | | `installer.clusterName` | string | `cluster-name` | The name given to your managed cluster in Calico Cloud. | | `installer.calicoCloudVersion` | string | `v22.1.0` | The version of Calico Cloud you're installing. | Example from values.yaml with clusterName and calicoCloudVersion ```yaml installer: clusterName: example-cluster calicoCloudVersion: v22.1.0 ``` ### Optional parameters for private registries[​](#optional-parameters-for-private-registries) If you're using a private registry, you must set the following parameters. | Parameter | Value | Example | Description | | ----------------------- | ------ | ------------------- | -------------------------------------------------------------------------------------------------------------------------- | | `installer.registry` | string | `my.registry/` | The name of your private registry. | | `installer.imagePath` | string | `path/to/directory` | The path to a directory in your private registry that contains images required to install Calico Cloud. | | `imagePullSecrets.name` | string | `secret-name` | The name of the image pull secret you will use to allow access from the `calico-cloud` namespace to your private registry. | ### Optional parameters for features[​](#optional-parameters-for-features) The following parameters enable certain features in Calico Cloud. These features can be enabled or disabled only by setting them in your `values.yaml` file at installation. | Feature name | Parameter | Values | | ------------------ | --------------------------------------------- | ------------------------------- | | Packet Capture | `installer.components.packetCaptureAPI.state` | `Enabled`, `Disabled` (default) | | Compliance Reports | `installer.components.compliance.enabled` | `true`, `false` (default) | > **SECONDARY:** If you're upgrading from Calico Cloud 19, the Packet Capture features will remain enabled unless you explicitly set them to `Disabled`. Use alternate feature keys for legacy features The Image Assurance and Container Threat Detection features were removed for new users in Calico Cloud 21.1.0. Legacy users of those features can continue to use a deprecated version until the features are completely removed in a future release. | Feature | Key | Values | | -------------------------- | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Image Assurance | `installer.components.imageAssurance.state` | `Enabled`, `Disabled` (default) | | Container Threat Detection | `installer.components.runtimeSecurity.state` | `Enabled`, `Disabled` (default\*) \* The default for new clusters is `Disabled`. For upgrades for previously connected clusters, the default will retain the previous state. | | Packet Capture | `installer.components.packetCaptureAPI.state` | `Enabled`, `Disabled` (default\*) \* The default for new clusters is `Disabled`. For upgrades for previously connected clusters, the default will retain the previous state. | | Compliance Reports | `installer.components.compliance.enabled` | `true`, `false` (default) | > **SECONDARY:** If you're upgrading from Calico Cloud 19, the Container Threat Detection and Packet Capture features will remain enabled unless you explicitly set them to `Disabled`. ### Optional parameters for pod scheduling and resource management[​](#optional-parameters-for-pod-scheduling-and-resource-management) For many Calico Cloud components, you can specify node selectors, tolerations, and resource requests and limits. The full list of Calico Cloud components is available in the default `values.yaml` file. > **SECONDARY:** Helm may overwrite previous customizations of custom resource fields available under the `installer.components` Helm parameter. For `installer.components`, you should define all your `values.yaml` customizations to be sure nothing is lost during Calico Cloud upgrades and reinstalls. ## Prepare your values.yaml with customizations[​](#prepare-your-valuesyaml-with-customizations) ***Prerequisites*** - You reviewed the information about available customizations in [About customizing your Helm installation](#about-customizing-your-helm-installation). - If you're installing from a private registry, you [added the Calico Cloud images to a private registry](https://docs.tigera.io/calico-cloud/get-started/setup-private-registry), and you have the following information about the registry: - Registry secret name > **SECONDARY:** If your private registry requires credentials, create a `calico-cloud` namespace on your cluster. Then, create an image pull secret and use this name for the **Registry Secret Name**. - Image registry - Image path 1. Add the Calico Cloud Helm repository to your local client: ```bash helm repo add calico-cloud https://installer.calicocloud.io/charts --force-update ``` 2. Save the default values definitions to your workstation so you can edit them locally: ```bash helm show values calico-cloud/calico-cloud > .yaml ``` All editable values are provided in the default values definitions. 3. Add values for the required parameters, `install.clusterName` and `install.calicoCloudVersion`. Example from values.yaml file with clusterName and calicoCloudVersion ```yaml installer: clusterName: example-cluster calicoCloudVersion: v22.1.0 ``` 4. Add values for the optional parameters. For each resource you want to edit, uncomment the object, add a value, and save. Example from values.yaml file with compliance reports disabled ```yaml installer: components: compliance: enabled: false ``` ## Install Calico Cloud as part of an automated workflow[​](#install-calico-cloud-as-part-of-an-automated-workflow-1) You can install Calico Cloud using repeatable kubectl or Helm commands together with valid client credentials. These commands can be added to any automated workflow. ***Prerequisites*** - You have generated a set of client credentials and you know the path to your secret. - You have a `values.yaml` file with your customizations. 1. Add the Calico Cloud Helm repository to your local client. ```bash helm repo add calico-cloud https://installer.calicocloud.io/charts --force-update ``` 2. Add the Calico Cloud custom resource definitions: ```bash helm upgrade --install calico-cloud-crds calico-cloud/calico-cloud-crds \ --namespace calico-cloud \ --create-namespace ``` 3. Apply the client credentials secret to your cluster. ```bash kubectl apply -f ``` > **INFO:** You should keep track of this with a secret management system. 4. Apply the Calico Cloud installer custom resource with your customizations in the `values.yaml` file. ```bash helm upgrade --install calico-cloud calico-cloud/calico-cloud \ --namespace calico-cloud \ -f .yaml ``` ## Additional resources[​](#additional-resources) - [Calico Cloud installation reference](https://docs.tigera.io/calico-cloud/reference/installation/api) ### Set up a private registry ## Big picture[​](#big-picture) Add Calico Cloud images to a private registry to install new clusters or update existing ones. ## Value[​](#value) In some deployments, installing Calico Cloud in clusters from third-party repos is not an option. Before you can install Calico Cloud from a private registry, you must first add Calico Cloud images to the registry. ## Concepts[​](#concepts) A **container image registry** (often known as a **registry**), is a service where you can push, pull, and store container images. In Kubernetes, a registry is considered *private* if it is not publicly available. A **private registry** is not publicly available. It must be accessible on a private network or with an **image pull secret** for authenticated access. An **image path** is a directory in a registry that contains images required to install Calico Cloud. ## Before you begin[​](#before-you-begin) **Required** - [Helm CLI](https://helm.sh/docs/intro/install/) command - Use the [Crane command](https://github.com/google/go-containerregistry/blob/main/cmd/crane/README.md) to follow steps in this guide. (Other tools can be used, but commands be adjusted accordingly.) - Image registry - Must be private (accessible only on a private network or using log in credentials) - Images copied to the registry must not be made publicly available - Push access to your registry ## How to[​](#how-to) Add the required Calico Cloud images to a private registry. ### Set up registry credentials[​](#set-up-registry-credentials) 1. Log into Calico Cloud and navigate to "Managed Clusters". 2. If your organization uses multiple projects to group managed clusters, click the **Project** menu and select the project you want your cluster to be part of. 3. Get the "Registry Credentials" by clicking on the icon. ![registry credentials](https://docs.tigera.io/assets/images/private-registry-icon-865a478b6480df31166ed0ad19c70167.png) 4. Apply the credentials so the Calico Cloud images can be accessed. ### Create the list of required images[​](#create-the-list-of-required-images) ### Select your version: v22.1.0 (latest)Helm command to add images for Calico Cloud v22.1.0 (latest) ```javascript helm repo add calico-cloud https://installer.calicocloud.io/charts INSTALLER_IMAGE="quay.io/tigera/cc-operator:$(helm show chart calico-cloud/calico-cloud | grep version: | sed -e 's/version: *//' -e 's/+/-g/')" IMAGES=( $INSTALLER_IMAGE quay.io/tigera/apiserver:v3.22.0-2.0 quay.io/tigera/compliance-benchmarker:v3.22.0-2.0 quay.io/tigera/compliance-controller:v3.22.0-2.0 quay.io/tigera/compliance-reporter:v3.22.0-2.0 quay.io/tigera/compliance-snapshotter:v3.22.0-2.0 quay.io/tigera/key-cert-provisioner:v3.22.0-2.0 quay.io/tigera/deep-packet-inspection:v3.22.0-2.0 quay.io/tigera/ui-apis:v3.22.0-2.0 quay.io/tigera/fluentd:v3.22.0-2.0 quay.io/tigera/fluentd-windows:v3.22.0-2.0 quay.io/tigera/guardian:v3.22.0-2.0 quay.io/tigera/intrusion-detection-controller:v3.22.0-2.0 quay.io/tigera/waf-http-filter:v3.22.0-2.0 quay.io/tigera/webhooks-processor:v3.22.0-2.0 quay.io/tigera/manager:v3.22.0-2.0 quay.io/tigera/packetcapture:v3.22.0-2.0 quay.io/tigera/policy-recommendation:v3.22.0-2.0 quay.io/tigera/egress-gateway:v3.22.0-2.0 quay.io/tigera/l7-collector:v3.22.0-2.0 quay.io/tigera/gateway-l7-collector:v3.22.0-2.0 quay.io/tigera/envoy:v3.22.0-2.0 quay.io/tigera/prometheus:v3.22.0-2.0 quay.io/tigera/prometheus-service:v3.22.0-2.0 quay.io/tigera/alertmanager:v3.22.0-2.0 quay.io/tigera/queryserver:v3.22.0-2.0 quay.io/tigera/kube-controllers:v3.22.0-2.0 quay.io/tigera/node:v3.22.0-2.0 quay.io/tigera/node-windows:v3.22.0-2.0 quay.io/tigera/typha:v3.22.0-2.0 quay.io/tigera/cni:v3.22.0-2.0 quay.io/tigera/cni-windows:v3.22.0-2.0 quay.io/tigera/es-gateway:v3.22.0-2.0 quay.io/tigera/linseed:v3.22.0-2.0 quay.io/tigera/dikastes:v3.22.0-2.0 quay.io/tigera/l7-admission-controller:v3.22.0-2.0 quay.io/tigera/pod2daemon-flexvol:v3.22.0-2.0 quay.io/tigera/csi:v3.22.0-2.0 quay.io/tigera/node-driver-registrar:v3.22.0-2.0 quay.io/tigera/envoy-gateway:v3.22.0-2.0 quay.io/tigera/envoy-proxy:v3.22.0-2.0 quay.io/tigera/envoy-ratelimit:v3.22.0-2.0 quay.io/tigera/operator:v1.40.1 quay.io/tigera/image-assurance-admission-controller:v1.22.9 quay.io/tigera/image-assurance-operator:v1.22.9 quay.io/tigera/image-assurance-container-runtime-adaptor:v1.22.9 quay.io/tigera/image-assurance-cluster-scanner:v1.22.9 quay.io/tigera/runtime-security-operator:v1.23.2 quay.io/tigera/skimble:v1.23.2 quay.io/tigera/cc-core:v0.3.3 quay.io/tigera/prometheus-operator:v3.22.0-2.0 quay.io/tigera/prometheus-config-reloader:v3.22.0-2.0 quay.io/tigera/cc-cni-config-scanner:v0.7 ) ``` ### Set up the private registry[​](#set-up-the-private-registry) ```bash REGISTRY= ``` If you want all images to come from the same path in your registry, set this image path value. Otherwise unset this environment variable or set it to the empty string. ```bash IMAGEPATH="" ``` #### Image examples[​](#image-examples) | Original image | Image Registry | Image Path | Private registry image | | ----------------------------------- | -------------- | ------------- | -------------------------------------------- | | `quay.io/tigera/typha:v1.2.3` | `my.registry/` | | `my.registry/tigera/typha:v1.2.3` | | `quay.io/tigera/typha:v1.2.3` | `my.registry/` | `custom-path` | `my.registry/custom-path/typha:v1.2.3` | | `quay.io/tigera/cc-operator:v4.5.6` | `my.registry/` | | `my.registry/tigera/cc-operator:v4.5.6` | | `quay.io/tigera/cc-operator:v4.5.6` | `my.registry/` | `custom-path` | `my.registry/custom-path/cc-operator:v4.5.6` | ### Copy images to your registry[​](#copy-images-to-your-registry) For Calico Cloud to install images from your registry, copy the images from the standard registries into your own registry. **Tab: Registry only** ```bash for image in ${IMAGES[@]}; do img_base=$(echo ${image} | sed "s#^.*/\([^/]*/[^/]*$\)#\1#") crane cp ${image} ${REGISTRY}${img_base} || break done ``` **Tab: Registry and ImagePath** ```bash for image in ${IMAGES[@]}; do img_base=$(echo ${image} | sed "s#^.*/##") crane cp ${image} ${REGISTRY}${IMAGEPATH}/${img_base} || break done ``` ## Install using the private registry[​](#install-using-the-private-registry) Follow the directions [to connect a cluster to Calico Cloud](https://docs.tigera.io/calico-cloud/get-started/install-cluster). ### Limitations and known issues for Windows nodes ## Calico Cloud feature limitations[​](#calico-cloud-feature-limitations) | Feature | Unsupported in this release | | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Platforms | - GKE | | Install and upgrade | - Typha component for scaling (Linux-based feature) | | Networking | - Overlay mode with BGP peering - IP in IP overlay with BGP routing - Cross-subnet support and MTU setting for VXLAN - IPv6 and dual stack - Dual-ToR - Service advertisement - Multiple networks to pods | | Policy | - Staged network-policy - Firewall integrations - Policy for hosts (host endpoints, including automatic host endpoints) - Tiered policy: TKG, GKE, AKS - WAF integration - AWS firewall integration - Fortinet integration | | Visibility and troubleshooting | - Packet capture - DNS logs - iptables logs - L7 logs | | Threat defense | - No threat defense features are supported. | | Image Assurance | - No Image Assurance features are supported. | | Multi-cluster management | - Multi-cluster management federated identity endpoints and services - Federated endpoint identity and services | | Compliance and security | - CIS benchmark and other reports - Wireguard encryption for pod-to-pod traffic and host-to-host traffic | | Data plane | - eBPF is a Linux-based feature | ## Calico Cloud BGP networking limitations[​](#calico-cloud-bgp-networking-limitations) If you are using Calico Cloud with BGP, note these current limitations with Windows. | Feature | Limitation | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | IP mobility/ borrowing | Calico Cloud IPAM allocates IPs to host in blocks for aggregation purposes. If the IP pool is full, nodes can also "borrow" IPs from another node's block. In BGP terms, the borrower then advertises a more specific "/32" route for the borrowed IP and traffic for that IP is only routed to the borrowing host. Windows nodes do not support this borrowing mechanism; they will not borrow IPs even if the IP pool is full and they mark their blocks so that Linux nodes will not borrow from them. | | IPs reserved for Windows | Calico Cloud IPAM allocates IPs in CIDR blocks. Due to networking requirements on Windows, four IPs per Windows node-owned block must be reserved for internal purposes. For example, with the default block size of /26, each block contains 64 IP addresses, 4 are reserved for Windows, leaving 60 for pod networking. To reduce the impact of these reservations, a larger block size can be configured at the IP pool scope (before any pods are created). | | Single IP block per host | Calico Cloud IPAM is designed to allocate blocks of IPs (default size /26) to hosts on demand. While the Calico Cloud CNI plugin was written to do the same, kube-proxy for Windows currently only supports a single IP block per host. To work around the default limit of one /26 per host there some options: - Use Calico Cloud BGP networking with the kubernetes datastore. In that mode, Calico Cloud IPAM is not used and the CNI host-local IPAM plugin is used with the node's Pod CIDR. To allow multiple IPAM blocks per host (at the expense of kube-proxy compatibility), set the `windows_use_single_network` flag to `false` in the `cni.conf.template` before installing Calico Cloud. Changing that setting after pods are networked is not recommended because it may leak HNS endpoints. | | IP-in-IP overlay | Calico Cloud's IPIP overlay mode cannot be used in clusters that contain Windows nodes because Windows does not support IP-in-IP. | | NAT-outgoing | Calico Cloud IP pools support a "NAT outgoing" setting with the following behaviour: - Traffic between Calico Cloud workloads (in any IP pools) is not NATted. - Traffic leaving the configured IP pools is NATted if the workload has an IP within an IP pool that has NAT outgoing enabled. Calico Cloud honors the above setting but it is only applied at pod creation time. If the IP pool configuration is updated after a pod is created, the pod's traffic will continue to be NATted (or not) as before. NAT policy for newly-networked pods will honor the new configuration. Calico Cloud automatically adds the host itself and its subnet to the NAT exclusion list. This behaviour can be disabled by setting flag `windows_disable_host_subnet_nat_exclusion` to `true` in `cni.conf.template` before running the install script. | | Service IP advertisement | This Calico Cloud feature is not supported on Windows. | ### Check your network configuration[​](#check-your-network-configuration) If you are using a networking type that requires layer 2 reachability (such as Calico Cloud with a BGP mesh and no peering to your fabric), you can check that your network has layer 2 reachability as follows: On each of your nodes, check the IP network of the network adapter that you plan to use for pod networking. For example, on Linux, assuming your network adapter is eth0, you can run: ```text $ ip addr show eth0 2: eth0: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 00:0c:29:cb:c8:19 brd ff:ff:ff:ff:ff:ff inet 192.168.171.136/24 brd 192.168.171.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fecb:c819/64 scope link valid_lft forever preferred_lft forever ``` In this case, the IPv4 is 192.168.171.136/24; which, after applying the /24 mask gives 192.168.171.0/24 for the IP network. Similarly, on Windows, you can run ```text PS C:\> ipconfig Windows IP Configuration Ethernet adapter vEthernet (Ethernet 2): Connection-specific DNS Suffix . : us-west-2.compute.internal Link-local IPv6 Address . . . . . : fe80::6d10:ccdd:bfbe:bce2%15 IPv4 Address. . . . . . . . . . . : 172.20.41.103 Subnet Mask . . . . . . . . . . . : 255.255.224.0 Default Gateway . . . . . . . . . : 172.20.32.1 ``` In this case, the IPv4 address is 172.20.41.103 and the mask is represented as bytes 255.255.224.0 rather than CIDR notation. Applying the mask, we get a network address 172.20.32.0/19. Because the Linux node has network 192.168.171.136/24 and the Windows node has a different network, 172.20.32.0/19, they are unlikely to be on the same layer 2 network. ## VXLAN networking limitations[​](#vxlan-networking-limitations) Because of differences between the Linux and Windows data plane feature sets, the following Calico Cloud features are not supported on Windows. | Feature | Limitation | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | IPs reserved for Windows | Calico Cloud IPAM allocates IPs in CIDR blocks. Due to networking requirements on Windows, four IPs per Windows node-owned block must be reserved for internal purposes. For example, with the default block size of /26, each block contains 64 IP addresses, 4 are reserved for Windows, leaving 60 for pod networking. To reduce the impact of these reservations, a larger block size can be configured at the IP pool scope (before any pods are created). | | Single IP block per host | Calico Cloud IPAM is designed to allocate blocks of IPs (default size /26) to hosts on demand. While the Calico Cloud CNI plugin was written to do the same, kube-proxy currently only supports a single IP block per host. To allow multiple IPAM blocks per host (at the expense of kube-proxy compatibility), set the `windows_use_single_network` flag to `false` in the `cni.conf.template` before installing Calico Cloud. Changing that setting after pods are networked is not recommended because it may leak HNS endpoints. | ## Routes are lost in cloud providers[​](#routes-are-lost-in-cloud-providers) If you create a Windows host with a cloud provider (AWS for example), the creation of the vSwitch at Calico Cloud install time can remove the cloud provider's metadata route. If your application relies on the metadata service, you may need to examine the routing table before and after installing Calico Cloud to reinstate any lost routes. ## VXLAN limitations[​](#vxlan-limitations) **VXLAN support** - Windows 1903 build 18317 and above - Windows 1809 build 17763 and above **Configuration updates** Certain configuration changes will not be honored after the first pod is networked. This is because Windows does not currently support updating the VXLAN subnet parameters after the network is created so updating those parameters requires the node to be drained: One example is the VXLAN VNI setting. To change such parameters: - Drain the node of all pods - Delete the Calico Cloud HNS network: ```powershell Import-Module -DisableNameChecking C:\TigeraCalico\libs\hns\hns.psm1 Get-HNSNetwork | ? Name -EQ "Calico Cloud" | Remove-HNSNetwork ``` - Update the configuration in `config.ps1`, run `uninstall-calico.ps1` and then `install-calico.ps1` to regenerate the CNI configuration. ## Pod-to-pod connections are dropped with TCP reset packets[​](#pod-to-pod-connections-are-dropped-with-tcp-reset-packets) Restarting Felix or changes to policy (including changes to endpoints referred to in policy) can cause pod-to-pod connections to be dropped with TCP reset packets when one of the following occurs: - The policy that applies to a pod is updated - Some ingress or egress policy that applies to a pod contains selectors and the set of endpoints that those selectors match changes Felix must reprogram the HNS ACL policy attached to the pod. This reprogramming can cause TCP resets. Microsoft has confirmed this is a HNS issue, and they are investigating. ## Service ClusterIPs incompatible with selectors on pod IPs in network policy[​](#service-clusterips-incompatible-with-selectors-on-pod-ips-in-network-policy) **Windows 1809 prior to build 17763.1432** On Windows nodes, kube-proxy unconditionally applies source NAT to traffic from local pods to service ClusterIPs. This means that, at the destination pod, where policy is applied, the traffic appears to come from the source host rather than the source pod. In turn, this means that a network policy with a source selector matching the source pod will not match the expected traffic. ## Network policy and using selectors[​](#network-policy-and-using-selectors) Under certain conditions, relatively simple Calico Cloud policies can require significant Windows data plane resources, that can cause significant CPU and memory usage, and large policy programming latency. We recommend avoiding policies that contain rules with both a source and destination selector. The following is an example of a policy that would be inefficient. The policy applies to all workloads, and it only allows traffic from workloads labeled as clients to workloads labeled as servers: ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: calico-dest-selector spec: selector: all() order: 500 ingress: - action: Allow destination: selector: role == "webserver" source: selector: role == "client" ``` Because the policy applies to all workloads, it will be rendered once per workload (even if the workload is not labeled as a server), and then the selectors will be expanded into many individual data plane rules to capture the allowed connectivity. Here is a much more efficient policy that still allows the same traffic: ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: calico-dest-selector spec: selector: role == "webserver" order: 500 ingress: - action: Allow source: selector: role == "client" ``` The destination selector is moved into the policy selector, so this policy is only rendered for workloads that have the `role: webserver` label. In addition, the rule is simplified so that it only matches on the source of the traffic. Depending on the number of webserver pods, this change can reduce the data plane resource usage by several orders of magnitude. ## Network policy with tiers[​](#network-policy-with-tiers) Because of the way the Windows data plane handles rules, the following limitations are required to avoid performance issues: - Tiers: maximum of 5 - `pass` rules: maximum of 10 per tier - If each tier contains a large number of rules, and has pass rules, you may need to reduce the number of tiers further. ## Flow log limitations[​](#flow-log-limitations) Calico Cloud supports flow logs with these limitations: - No packet/bytes stats for denied traffics - Inaccurate `num_flows_started` and `num_flows_completed` stats with VXLAN networking - No DNS stats - No Http stats - No RuleTrace for tiers - No BGP logs ## DNS Policy limitations[​](#dns-policy-limitations) > **SECONDARY:** DNS Policy is a tech preview feature. Tech preview features may be subject to significant changes before they become GA. Calico Cloud supports DNS policy on Windows with these limitations: - It could take up to 5 seconds for the first TCP SYN packet to go through, for a connection to a DNS domain name. This is because DNS policies are dynamically programmed. The first TCP packet could be dropped since there is no policy to allow it until Calico Cloud detects domain IPs from DNS response and programs DNS policy rules. The Windows TCPIP stack will send SYN again after TCP Retransmission timeout (RTO) if previous SYN has been dropped. - Some runtime libraries do not honour DNS TTL. Instead, they manage their own DNS cache which has a different TTL value for DNS entries. On .NET Framework, the value to control DNS TTL is ServicePointManager.DnsRefreshTimeout which has default value of 120 seconds - [DNS refresh timeout](https://docs.microsoft.com/en-us/dotnet/api/system.net.servicepointmanager.dnsrefreshtimeout). It is important that Calico Cloud uses a longer TTL value than the one used by the application, so that DNS policy will be in place when the application is making outbound connections. The configuration item “WindowsDNSExtraTTL” should have a value bigger than the maximum value of DNS TTL used by the runtime libraries for your applications. - Due to the limitations of Windows container networking, a policy update could have an impact on performance. Programming DNS policy may result in more policy updates. Setting “WindowsDNSExtraTTL” to a bigger number will reduce the performance impact. ### Troubleshooting checklist ## Check Calico Cloud installation[​](#check-calico-cloud-installation) Installing Calico Cloud on your Kubernetes cluster is managed by the Calico Cloud operator. The Calico Cloud operator is deployed as a Deployment in the `calico-cloud` namespace, and records status in a custom resource named `installer`. Check the `installer` status using the following command. ```bash kubectl get installer default --namespace calico-cloud -o jsonpath --template '{.status}' ``` **Sample output** ```text {"clusterName":"my-new-cluster","state":"installing"} ``` After `state` is `complete`, Calico Cloud is properly installed. ## Check logs for fatal errors[​](#check-logs-for-fatal-errors) Check that the Calico Cloud operator is running and that logs do not have any fatal errors. ```bash kubectl logs -n calico-cloud deployment/calico-cloud-controller-manager 2022-04-04T14:34:32.472Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": "127.0.0.1:8080"} 2022-04-04T14:34:32.472Z INFO setup starting manager 2022-04-04T14:34:32.472Z INFO setup config {"ccBaseURL": "https://www.dev.calicocloud.io/api", "debug": false, "leader-elect": true} I0404 14:34:32.472586 1 leaderelection.go:243] attempting to acquire leader lease calico-cloud/c2ad41ce.calicocloud.io... 2022-04-04T14:34:32.472Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"} I0404 14:34:32.480870 1 leaderelection.go:253] successfully acquired lease calico-cloud/c2ad41ce.calicocloud.io <...> ``` ## Check custom resources[​](#check-custom-resources) Verify that you have the **installer custom resource**, and that the values are appropriate for your environment. ```bash kubectl get installers.operator.calicocloud.io --namespace calico-cloud -o yaml ``` ```yaml apiVersion: v1 items: - apiVersion: operator.calicocloud.io/v1 kind: Installer metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"operator.calicocloud.io/v1","kind":"Installer","metadata":{"annotations":{},"name":"my-new-cluster","namespace":"calico-cloud"}} creationTimestamp: '2022-04-04T14:34:29Z' generation: 1 name: my-new-cluster namespace: calico-cloud resourceVersion: '1102' uid: eb1d1cd0-f01f-47b2-81fe-8eee46dbe712 status: clusterName: my-new-cluster message: '' resourceVersion: '' state: installing kind: List metadata: resourceVersion: '' selfLink: '' ``` ## Send failed installation diagnostics to Calico Cloud support[​](#send-failed-installation-diagnostics-to-calico-cloud-support) To upload diagnostics about a failed installation to Calico Cloud support, run the following command: ```bash kubectl patch installer -n calico-cloud default --type merge -p='{"spec":{"uploadDiags":true}}' ``` ### Tigera Operator troubleshooting checklist If you have issues getting your cluster up and running, use this checklist. - [Check installation start errors](#check-installation-start-errors) - [Check Calico Cloud installation](#check-calico-cloud-installation) - [Check logs for fatal errors](#check-logs-for-fatal-errors) - [Check custom resources](#check-custom-resources) - [Check pod capacity](#check-pod-capacity) - [Check the web console dashboard for traffic](#check-manager-ui-dashboard-for-traffic) ## Check installation start errors[​](#check-installation-start-errors) Are you seeing any of these issues at the start of installation? ### \[ERROR] Detected plugin ls: No such file or directory, it is currently not supported[​](#error-detected-plugin-ls-no-such-file-or-directory-it-is-currently-not-supported) The cluster you are using to install Calico Cloud does not have a CNI plugin installed, the CNI is incompatible. If your cluster has functional pod networking and you see this message, it is likely that kubelet has been configured to use kubenet networking, which is not compatible with Calico Cloud. You can use a different cluster, or re-create your cluster with [compatible networking](https://docs.tigera.io/calico-cloud/get-started/system-requirements). ### Calico Cloud cannot be connected to a cluster with FIPS mode enabled[​](#calico-cloud-cannot-be-connected-to-a-cluster-with-fips-mode-enabled) At this time, FIPS mode is not supported in Calico Cloud. Disable FIPS mode in the cluster and install again. ### Install script is taking a long time[​](#install-script-is-taking-a-long-time) If you are migrating a large cluster from a previous manifest-based Calico install, the script can take some time; this is normal. But, it could also mean that your cluster has an incompatibility. Go to the next step [Check Calico Cloud installation](#check-calico-cloud-installation). ## Check Calico Cloud installation[​](#check-calico-cloud-installation) Installing Calico Cloud on your Kubernetes cluster is managed by the Tigera Operator. The Tigera Operator is deployed as a ReplicaSet in the `tigera-operator` namespace, and records status in a custom resource named, `tigerastatus`. The operator get its configuration from several Custom Resources (CRs); the central one is the Installation CR. Check `tigerastatus` using the following command. ```bash kubectl get tigerastatus ``` **Sample output** ```text NAME AVAILABLE PROGRESSING DEGRADED SINCE apiserver True False False 10m calico True False False 11m cloud-core True False False 11m compliance True False False 9m39s intrusion-detection True False False 9m49s log-collector True False False 9m29s management-cluster-connection True False False 9m54s monitor True False False 10m runtime-security True False False 10m ``` If all components show a status of "Available" = TRUE, Calico Cloud is properly installed. > **SECONDARY:** The `runtime-security` component is available only if [the container threat detection feature is enabled](https://docs.tigera.io/calico-cloud/threat/container-threat-detection#enable-container-threat-detection). **Issue: Calico Cloud is not installed** If Calico Cloud is not installed, you'll get the following error. Install Calico Cloud on the node using the `curl` command that you got from Support. ```bash kubectl get tigerastatus error: the server doesn't have a resource type "tigerastatus" ``` **Issue: Calico Cloud components are missing or are degraded** If some of the Calico Cloud components are Available = FALSE or DEGRADED = TRUE, run the following command and contact Support with the following output. ```bash kubectl get tigerastatus -o yaml ``` > **SECONDARY:** If you are using the **AWS or Azure CNI plugin**, a degraded state is likely because you do not have enough pod capacity on your nodes. To fix this, see [Check pod capacity](#check-pod-capacity). **Sample output** In the following example, the typha component has an issue because it is showing `AVAILABLE: FALSE`, and `DEGRADED: TRUE`. To understand details of Calico Cloud components, see [Deep dive into custom resources](#deep-dive-into-custom-resources). ```yaml apiVersion: v1 items: - apiVersion: operator.tigera.io/v1 kind: TigeraStatus metadata: creationTimestamp: '2020-12-30T17:13:30Z' generation: 1 managedFields: - apiVersion: operator.tigera.io/v1 fieldsType: FieldsV1 fieldsV1: f:spec: {} f:status: .: {} f:conditions: {} manager: operator operation: Update time: '2020-12-30T17:16:20Z' name: calico resourceVersion: '8166' selfLink: /apis/operator.tigera.io/v1/tigerastatuses/calico uid: 39a8a2d0-2074-418c-b52d-0baa0a48f4a1 spec: {} status: conditions: - lastTransitionTime: '2020-12-30T17:13:30Z' status: 'False' type: Available - lastTransitionTime: '2020-12-30T17:13:30Z' message: DaemonSet "calico-system/calico-node" is not yet scheduled on any nodes reason: Not all pods are ready status: 'True' type: Progressing - lastTransitionTime: '2020-12-30T17:13:30Z' message: 'failed to wait for operator typha deployment to be ready: waiting for typha to have 4 replicas, currently at 3' reason: error migrating resources to calico-system status: 'True' type: Degraded kind: List metadata: resourceVersion: '' selfLink: '' ``` ## Check logs for fatal errors[​](#check-logs-for-fatal-errors) Check that the Tigera Operator is running and that logs do not have any fatal errors. ```bash kubectl get pods -n tigera-operator ``` ```text NAME READY STATUS RESTARTS AGE tigera-operator-8687585b66-68gmr 1/1 Running 0 139m ``` ```bash kubectl logs -n tigera-operator tigera-operator-8687585b66-68gmr ``` ```text 2020/12/30 17:38:54 [INFO] Version: 90975f4 2020/12/30 17:38:54 [INFO] Go Version: go1.14.4 2020/12/30 17:38:54 [INFO] Go OS/Arch: linux/amd64 {"level":"info","ts":1609349935.2848425,"logger":"setup","msg":"Checking type of cluster","provider":""} {"level":"info","ts":1609349935.2868738,"logger":"setup","msg":"Checking if TSEE controllers are required","required":true} <...> ``` ## Check custom resources[​](#check-custom-resources) Verify that you have the **installation custom resource**, and that the values are appropriate for your environment. ```bash kubectl get installation.operator.tigera.io default -o yaml ``` ```yaml apiVersion: operator.tigera.io/v1 kind: Installation metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"operator.tigera.io/v1","kind":"Installation","metadata":{"annotations":{},"name":"default"},"spec":{"imagePullSecrets":[{"name":"tigera-pull-secret"}],"variant":"TigeraSecureEnterprise"}} creationTimestamp: '2021-01-20T19:50:23Z' generation: 2 managedFields: - apiVersion: operator.tigera.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:kubectl.kubernetes.io/last-applied-configuration: {} f:spec: .: {} f:imagePullSecrets: {} f:variant: {} manager: kubectl operation: Update time: '2021-01-20T19:50:23Z' - apiVersion: operator.tigera.io/v1 fieldsType: FieldsV1 fieldsV1: f:spec: f:calicoNetwork: .: {} f:bgp: {} f:hostPorts: {} f:ipPools: {} f:mtu: {} f:multiInterfaceMode: {} f:nodeAddressAutodetectionV4: .: {} f:firstFound: {} f:cni: .: {} f:ipam: .: {} f:type: {} f:type: {} f:componentResources: {} f:flexVolumePath: {} f:nodeUpdateStrategy: .: {} f:rollingUpdate: .: {} f:maxUnavailable: {} f:type: {} f:status: .: {} f:variant: {} manager: operator operation: Update time: '2021-01-20T19:55:10Z' name: default resourceVersion: '5195' selfLink: /apis/operator.tigera.io/v1/installations/default uid: 016c3f0b-39f0-48a0-9da8-a59a81ed9128 spec: calicoNetwork: bgp: Enabled hostPorts: Enabled ipPools: - blockSize: 26 cidr: 10.42.0.0/16 encapsulation: IPIP natOutgoing: Enabled nodeSelector: all() mtu: 0 multiInterfaceMode: None nodeAddressAutodetectionV4: firstFound: true cni: ipam: type: Calico type: Calico componentResources: - componentName: Node resourceRequirements: requests: cpu: 250m flexVolumePath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds imagePullSecrets: - name: tigera-pull-secret nodeUpdateStrategy: rollingUpdate: maxUnavailable: 1 type: RollingUpdate variant: TigeraSecureEnterprise status: variant: TigeraSecureEnterprise ``` Verify that you have the following custom resources. In the default installation, there is no configuration information. **Check API server** ```bash kubectl get apiserver.operator.tigera.io tigera-secure ``` ```text NAME AGE tigera-secure 85m ``` **Check cloud core** ```bash kubectl get cloudcore.operator.tigera.io tigera-secure ``` ```text NAME AGE tigera-secure 88m ``` **Check compliance** ```bash kubectl get compliance.operator.tigera.io tigera-secure ``` ```text NAME AGE tigera-secure 90m ``` **Check intrusion detection** ```bash kubectl get intrusiondetection.operator.tigera.io tigera-secure ``` ```text NAME AGE tigera-secure 93m ``` **Check log collector** ```bash kubectl get logcollector.operator.tigera.io tigera-secure ``` ```text NAME AGE tigera-secure 96m ``` **Check management cluster** ```bash kubectl get ManagementClusterConnection.operator.tigera.io tigera-secure -o yaml ``` ```yaml apiVersion: operator.tigera.io/v1 kind: ManagementClusterConnection metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"operator.tigera.io/v1","kind":"ManagementClusterConnection","metadata":{"annotations":{},"name":"tigera-secure"},"spec":{"managementClusterAddr":".tigera.io:9000"}} creationTimestamp: '2021-01-20T19:55:40Z' generation: 1 managedFields: - apiVersion: operator.tigera.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:kubectl.kubernetes.io/last-applied-configuration: {} f:spec: .: {} f:managementClusterAddr: {} manager: kubectl operation: Update time: '2021-01-20T19:55:40Z' name: tigera-secure resourceVersion: '5425' selfLink: /apis/operator.tigera.io/v1/managementclusterconnections/tigera-secure uid: b7a2093e-a4b6-4e76-b291-15f45bfa11cf spec: managementClusterAddr: .tigera.io:9000 ``` **Check monitor** ```bash kubectl get monitor.operator.tigera.io tigera-secure ``` ```text NAME AGE tigera-secure 98m ``` \*\*Check runtime security \*\* ```bash kubectl get runtimesecurity.operator.tigera.io default ``` ```text NAME AGE default 99m ``` > **SECONDARY:** The `runtime-security` custom resource will only be available if the container threat detection feature is enabled. For more information on operator custom resources see the [Installation API reference](https://docs.tigera.io/calico-cloud/reference/installation/api). ### Deep dive into custom resources[​](#deep-dive-into-custom-resources) Run the following command to see if you have required custom resources: ```bash kubectl get tigerastatus ``` | | NAME | AVAILABLE | PROGRESSING | DEGRADED | SINCE | | - | ----------------------------- | --------- | ----------- | -------- | ----- | | 1 | apiserver | TRUE | FALSE | FALSE | 10m | | 2 | calico | TRUE | FALSE | FALSE | 11m | | 3 | cloud-core | TRUE | FALSE | FALSE | 11m | | 4 | compliance | TRUE | FALSE | FALSE | 9m39s | | 5 | intrusion-detection | TRUE | FALSE | FALSE | 9m49s | | 6 | log-collector | TRUE | FALSE | FALSE | 9m29s | | 7 | management-cluster-connection | TRUE | FALSE | FALSE | 9m54s | | 8 | monitor | TRUE | FALSE | FALSE | 11m | | 9 | runtime-security | TRUE | FALSE | FALSE | 10m | **1 - api server** `apiserver` is a required component that is an aggregated api-server. It is required for things like applying the tigera license. If `tigerastatus` reports it as unavailable or degraded, check the pods and logs in the `calico-system`namespace. For example, ```bash kubectl get pods -n calico-system ``` ```text NAME READY STATUS RESTARTS AGE calico-apiserver-5c75bc8d4b-sbn6g 2/2 Running 0 45m ``` **2 - calico** `calico` is the core component for networking. If it is not available or degraded, check the pods and their logs in the `calico-system` namespace. There should be a calico-node pod running on each of your nodes. You should have at least one `calico-typha` pod and the number will scale with the number of nodes in your cluster. You should have a `calico-kube-controllers` pod running. For example, ```bash kubectl get pods -n calico-system ``` ```text NAME READY STATUS RESTARTS AGE calico-kube-controllers-5c77d4d559-hfl5d 1/1 Running 0 44m calico-node-6s2c9 1/1 Running 0 40m calico-node-8nf28 1/1 Running 0 41m calico-node-djlrg 1/1 Running 0 40m calico-node-ms8nv 1/1 Running 0 40m calico-node-t7pck 1/1 Running 0 40m calico-typha-bdb494458-76gcx 1/1 Running 0 41m calico-typha-bdb494458-847tr 1/1 Running 0 41m calico-typha-bdb494458-k8lhj 1/1 Running 0 40m calico-typha-bdb494458-vjbjz 1/1 Running 0 40m ``` **3 - cloud-core** `cloud-core` is responsible for predefined and custom roles for users. Check the pods and logs in the `calico-cloud` namespace with the label selector `k8s-app=cc-core-operator`. ```bash $ kubectl get pods -n calico-cloud -l k8s-app=cc-core-operator ``` ```text NAME READY STATUS RESTARTS AGE cc-core-operator-126dcd494a-9kj7g 1/1 Running 0 80m ``` **4 - compliance** `compliance` is responsible for the compliance features. Check the pods and logs in the `tigera-compliance` namespace. ```bash $ kubectl get pods -n tigera-compliance ``` ```text NAME READY STATUS RESTARTS AGE compliance-benchmarker-bqvps 1/1 Running 0 65m compliance-benchmarker-h58hr 1/1 Running 0 65m compliance-benchmarker-kdtwp 1/1 Running 0 65m compliance-benchmarker-mzm2z 1/1 Running 0 65m compliance-benchmarker-s5mmf 1/1 Running 0 65m compliance-controller-77785646df-ws2cj 1/1 Running 0 65m compliance-snapshotter-6bcbdc65b-66k9v 1/1 Running 0 65m ``` **5 - intrusion-detection** `intrusion-detection` is responsible for the intrusion detection features. Check the pods and logs in the `tigera-intrusion-detection` namespace. ```bash $ kubectl get pods -n tigera-intrusion-detection ``` ```text NAME READY STATUS RESTARTS AGE intrusion-detection-controller-669bf45c75-grvz9 1/1 Running 0 66m intrusion-detection-es-job-installer-xm22v 1/1 Running 0 66m ``` **6 - log-collector** `log-collector` collects flow and other logs and forwards them to Calico Cloud. Check the pods and logs in the `tigera-fluentd` namespace. You should have one pod running on each of your nodes. ```bash kubectl get pods -n tigera-fluentd ``` ```text NAME READY STATUS RESTARTS AGE fluentd-node-5mzh6 1/1 Running 0 70m fluentd-node-7vmxw 1/1 Running 0 70m fluentd-node-bbc4p 1/1 Running 0 70m fluentd-node-chfz4 1/1 Running 0 70m fluentd-node-d6f56 1/1 Running 0 70m ``` **7 - management-cluster-connection** The `management-cluster-connection` is required for your managed clusters to connect to the Calico Cloud backend. If it is not available or degraded, check the pods and logs in the `tigera-guardian` namespace. ```bash kubectl get pods -n tigera-guardian ``` ```text NAME READY STATUS RESTARTS AGE tigera-guardian-7d5d94d5cc-49rg8 1/1 Running 0 48m ``` To verify that the guardian component has network connectivity to the management cluster: Find the URL to the management cluster: ```bash kubectl get managementclusterconnection tigera-secure -o=jsonpath='{.spec.managementClusterAddr}' .tigera.io:9000 ``` then, from a worker node, verify network connectivity to the management cluster: ```bash openssl s_client -connect .tigera.io:9000 ``` ```text CONNECTED(00000003) depth=0 CN = tigera-voltron verify error:num=18:self signed certificate verify return:1 depth=0 CN = tigera-voltron verify return:1 --- Certificate chain 0 s:CN = tigera-voltron i:CN = tigera-voltron --- Server certificate -----BEGIN CERTIFICATE----- MIIC5DCCAcygAwIBAgIBATANBgkqhkiG9w0BAQsFADAZMRcwFQYDVQQDEw50aWdl cmEtdm9sdHJvbjAeFw0yMDEyMjExOTA1MzhaFw0yNTEyMjAxOTA1MzhaMBkxFzAV <...> ``` **8 - monitor** `monitor` is responsible for configuring prometheus and associated custom resources. Check the pods and logs in the `tigera-prometheus` namespace. ```bash $ kubectl get pods -n tigera-prometheus ``` ```text NAME READY STATUS RESTARTS AGE alertmanager-calico-node-alertmanager-0 2/2 Running 0 125m alertmanager-calico-node-alertmanager-1 2/2 Running 0 125m alertmanager-calico-node-alertmanager-2 2/2 Running 0 125m calico-prometheus-operator-77bf897c9b-7f88x 1/1 Running 0 125m prometheus-calico-node-prometheus-0 3/3 Running 1 125m ``` **9 - runtime-security** `runtime-security` is responsible for the container threat detection feature. Check the pods and logs in the `calico-cloud` namespace with the label selector `k8s-app=tigera-runtime-security-operator`. ```bash $ kubectl get pods -n calico-cloud -l k8s-app=tigera-runtime-security-operator ``` ```text NAME READY STATUS RESTARTS AGE tigera-runtime-security-operator-127b606afc-ap25k 1/1 Running 0 80m ``` ### Check additional custom resources[​](#check-additional-custom-resources) Check for the presence of other custom resources created by the Tigera Operator: FelixConfiguration, IPPool, Tigera License, and Prometheus for component metrics. `FelixConfiguration` contains configuration that are not configured as environment variables to the`calico-node` container. ```bash kubectl get Felixconfiguration default ``` ```text NAME CREATED AT default 2021-01-20T19:49:35Z ``` The operator creates a default `IPPool` for your pod networking if it does not already exist; in this case, the CIDR is taken from the Installation CR. ```bash kubectl get IPPool ``` ```text NAME CREATED AT default-ipv4-ippool 2021-01-20T19:49:35Z ``` A Tigera license is applied by the installation script. ```bash kubectl get licensekeys.projectcalico.org ``` ```text NAME AGE default 120m ``` The installation script deploys a Prometheus operator and associated custom resources. If you already have a Prometheus operator running in your cluster, contact Tigera support. ```bash kubectl get pods -n tigera-prometheus ``` ```text NAME READY STATUS RESTARTS AGE alertmanager-calico-node-alertmanager-0 2/2 Running 0 125m alertmanager-calico-node-alertmanager-1 2/2 Running 0 125m alertmanager-calico-node-alertmanager-2 2/2 Running 0 125m calico-prometheus-operator-77bf897c9b-7f88x 1/1 Running 0 125m prometheus-calico-node-prometheus-0 3/3 Running 1 125m ``` ## Check pod capacity[​](#check-pod-capacity) If cluster does not have enough capacity, it will not be able to deploy pods. There is no specific error associated with this condition. The high-level components Calico Cloud needs to run are: - Per node: 1 fluentd, 1 compliance benchmarker - On top of per node: 3 alertmanager (from statefulset), 1 prometheus, 1 prometheus operator, 1 kube-controllers, 2 compliance snapshotter and controller, 1 guardian, 1 ids controller, 1 apiserver Some clusters have limited pod-networked pod capacity. - For the AWS CNI plugin, the [number of pods that can be networked is based on the size of the instance](https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html). - For AKS with the Azure CNI, the [number of pods that can be networked is set at cluster deployment time or when new node pools are created (default of 30)](https://docs.microsoft.com/en-us/azure/aks/configure-azure-cni#maximum-pods-per-node). - For GKE, there is a [hard limit of 110 pods per node](https://cloud.google.com/kubernetes-engine/docs/best-practices/scalability#dimension_limits). - For Calico CNI, the pod limit is based on the available IPs in the IPPool, and there is no specific per node limit. Verify you have the following pod-networked pod capacity. - Verify on each node in your cluster that there is capacity for at least 2 pods. - Verify there is capacity for at least 11 pods in the cluster in addition to the per node capacity. To check the capacity of individual nodes on AWS or AKS, query the node status and look at `Capacity.Pods` (which is the total capacity for the node). To get the number of pod-networked pods for a node, count the pods on the node that are pod-networked (non-hostNetworked pods). ## Check pod security policy violation[​](#check-pod-security-policy-violation) If your cluster is using Kubernetes version 1.24 or earlier, a pod security policy (PSP) violation may be blocking pods on the cluster. Search for the term `PodSecurityPolicy` in the status message of failed cluster deployments. If a PSP is present, install open source Calico in the cluster before you connect to Calico Cloud. ## Check the web console dashboard for traffic[​](#check-the-web-console-dashboard-for-traffic) ### Web console main dashboard is missing traffic graphs[​](#web-console-main-dashboard-is-missing-traffic-graphs) When you log in to the web console, the first item in the left nav is the main Dashboard. This dashboard is a birds-eye view of your managed cluster activity for policy and networking. For the graphs to display traffic, you must have the Prometheus operator. If it is missing, the following message is displayed during installation: `Prometheus Operator detected in the cluster. Skipping Tigera Prometheus Operator` To install an appropriate Prometheus operator, contact Support below. ### Upgrade Calico Cloud To upgrade a managed cluster to the latest version of Calico Cloud: 1. From the web console, go to the **Managed Clusters** page. 2. If your organization uses multiple projects to group managed clusters, click the **Project** menu and select the project you want your cluster to be part of. 3. For the cluster you want to upgrade, select **Actions** > **Reinstall**. 4. In the **Reinstall Cluster** dialog, select a newer version of Calico Cloud from the list, click **Reinstall**, and copy the generated kubectl command. Use alternate manifest for legacy features The Image Assurance and Container Threat Detection features were removed for new users in Calico Cloud 21.1.0. Legacy users of those features can continue to use a deprecated version until the features are completely removed in a future release. To continue using these features, modify the generated command by replacing **two instances** of `deploy.yaml` with `deploy-with-container-security.yaml`. This change gives you a manifest with all three legacy features enabled. You cannot enable or disable these features individually. Example of generated kubectl command with alternate manifest ```bash kubectl apply -f https://installer.calicocloud.io/manifests/cc-operator/latest/deploy-with-container-security.yaml && curl -H "Authorization: Bearer ..." "https://www.calicocloud.io/api/managed-cluster/deploy-with-container-security.yaml?version=v22.1.0" | kubectl apply -f - ``` 5. From a terminal, paste and run the command. The cluster's status under **Connection Status** changes to **Disconnected: Installing**. When the upgrade is complete, the status changes to **Connected**. --- ## Users ### Users Manage users and user permissions for the web console. ##### [Set up users](https://docs.tigera.io/calico-cloud/users/user-management) [Authenticate and authorize users.](https://docs.tigera.io/calico-cloud/users/user-management) ##### [Creating and assigning custom roles](https://docs.tigera.io/calico-cloud/users/create-and-assign-custom-roles) [Create and assign custom roles to give specific, cluster-specific permission to your users.](https://docs.tigera.io/calico-cloud/users/create-and-assign-custom-roles) ##### [Assign roles to Entra ID groups](https://docs.tigera.io/calico-cloud/users/create-custom-role-for-entra-id-group) [Create custom roles for Entra\ ID groups.](https://docs.tigera.io/calico-cloud/users/create-custom-role-for-entra-id-group) ### Set up users ## Authentication[​](#authentication) Calico Cloud supports Google Social login and username / password for user authentication. ## Roles and authorization[​](#roles-and-authorization) Users can have one or more of the following predefined user roles to access features in the web console. The default permissions align with typical needs for each role. ### Owner[​](#owner) The Owner role has the highest level of access and typically corresponds to the account creator. > **SECONDARY:** The Owner role cannot be assigned to new users. The only Owner is the user who created the Calico Cloud account. | Feature | Permission Level | | --------------------------------- | ------------------ | | Service Graph and Flow Visualizer | view | | Policies | view, edit | | Nodes and Endpoints | view | | Network Sets | view, edit | | Managed Clusters | view, edit, delete | | Compliance Reports | view | | Timeline | view | | Alerts | view, edit | | Kibana | view, edit | | Image Assurance | view, edit | | Manage Team | view, edit | | Usage Metrics | view | | Threat Feeds | view, edit | | Web Application Firewall | view, edit | | Container Threat Detection | view, edit | | Dashboards | view, edit | ### Admin[​](#admin) The Admin role provides broad administrative access for day-to-day configuration and management of Calico Cloud. | Feature | Permission Level | | --------------------------------- | ------------------ | | Service Graph and Flow Visualizer | view | | Policies | view, edit | | Nodes and Endpoints | view | | Network Sets | view, edit | | Managed Clusters | view, edit, delete | | Compliance Reports | view | | Timeline | view | | Alerts | view, edit | | Kibana | view, edit | | Image Assurance | view, edit | | Manage Team | view, edit | | Usage Metrics | - | | Threat Feeds | view, edit | | Web Application Firewall | view, edit | | Container Threat Detection | view, edit | | Dashboards | view, edit | ### User Admin[​](#user-admin) The User Admin role has the ability to manage team members and their assigned roles. | Feature | Permission Level | | --------------------------------- | ---------------- | | Service Graph and Flow Visualizer | - | | Policies | - | | Nodes and Endpoints | - | | Network Sets | - | | Managed Clusters | - | | Compliance Reports | - | | Timeline | - | | Alerts | - | | Kibana | - | | Image Assurance | - | | Manage Team | view, edit | | Usage Metrics | - | | Threat Feeds | - | | Web Application Firewall | - | | Container Threat Detection | - | | Dashboards | - | ### Cluster Connection Admin[​](#cluster-connection-admin) The Cluster Connection Admin role has administrative capabilities of managed clusters. | Feature | Permission Level | | --------------------------------- | ------------------ | | Service Graph and Flow Visualizer | - | | Policies | - | | Nodes and Endpoints | - | | Network Sets | - | | Managed Clusters | view, edit, delete | | Compliance Reports | - | | Timeline | - | | Alerts | - | | Kibana | - | | Image Assurance | - | | Manage Team | - | | Usage Metrics | - | | Threat Feeds | - | | Web Application Firewall | - | | Container Threat Detection | - | | Dashboards | - | ### Viewer[​](#viewer) The Viewer role provides read-only access to most operational and configuration data within Calico Cloud. Ideal for users who need visibility without making changes. | Feature | Permission Level | | --------------------------------- | ---------------- | | Service Graph and Flow Visualizer | view | | Policies | view | | Nodes and Endpoints | view | | Network Sets | view | | Managed Clusters | view | | Compliance Reports | view | | Timeline | view | | Alerts | view | | Kibana | view | | Image Assurance | - | | Manage Team | view | | Usage Metrics | - | | Threat Feeds | view | | Web Application Firewall | view | | Container Threat Detection | view | | Dashboards | view | ### DevOps[​](#devops) The DevOps role is designed for users responsible for application deployment, CI/CD integration, and managing network policies and configurations relevant to their applications. | Feature | Permission Level | | --------------------------------- | ---------------- | | Service Graph and Flow Visualizer | view | | Policies | view, edit | | Nodes and Endpoints | view | | Network Sets | view, edit | | Managed Clusters | view, edit | | Compliance Reports | - | | Timeline | view | | Alerts | view, edit | | Kibana | view, edit | | Image Assurance | view, edit | | Manage Team | view | | Usage Metrics | - | | Threat Feeds | view, edit | | Web Application Firewall | view | | Container Threat Detection | view | | Dashboards | view | ### Security[​](#security) The Security role focuses on security posture management, including policy definition, threat monitoring, vulnerability management (Image Assurance), and incident response. | Feature | Permission Level | | --------------------------------- | ---------------- | | Service Graph and Flow Visualizer | view | | Policies | view, edit | | Nodes and Endpoints | view | | Network Sets | view, edit | | Managed Clusters | view | | Compliance Reports | view | | Timeline | view | | Alerts | view, edit | | Kibana | view, edit | | Image Assurance | view, edit | | Manage Team | view | | Usage Metrics | - | | Threat Feeds | view, edit | | Web Application Firewall | view, edit | | Container Threat Detection | view, edit | | Dashboards | view | ### Compliance[​](#compliance) The Compliance role provides focused access to compliance reporting and related policy information, suitable for auditors or compliance officers. | Feature | Permission Level | | --------------------------------- | ---------------- | | Service Graph and Flow Visualizer | - | | Policies | view | | Nodes and Endpoints | view | | Network Sets | - | | Managed Clusters | - | | Compliance Reports | view | | Timeline | - | | Alerts | - | | Kibana | - | | Image Assurance | - | | Manage Team | - | | Usage Metrics | - | | Threat Feeds | - | | Web Application Firewall | - | | Container Threat Detection | - | | Dashboards | - | ### Usage Metrics[​](#usage-metrics) This role grants specific access to view usage metrics for the Calico Cloud account. | Feature | Permission Level | | --------------------------------- | ---------------- | | Service Graph and Flow Visualizer | - | | Policies | - | | Nodes and Endpoints | - | | Network Sets | - | | Managed Clusters | - | | Compliance Reports | - | | Timeline | - | | Alerts | - | | Kibana | - | | Image Assurance | - | | Manage Team | - | | Usage Metrics | view | | Threat Feeds | - | | Web Application Firewall | - | | Container Threat Detection | - | | Dashboards | - | ### Image Assurance Admin[​](#image-assurance-admin) This role provides administrative control specifically over the Image Assurance feature, including configuring registries, policies, and viewing scan results. | Feature | Permission Level | | --------------------------------- | ---------------- | | Service Graph and Flow Visualizer | - | | Policies | - | | Nodes and Endpoints | - | | Network Sets | - | | Managed Clusters | - | | Compliance Reports | - | | Timeline | - | | Alerts | - | | Kibana | - | | Image Assurance | view, edit | | Manage Team | - | | Usage Metrics | - | | Threat Feeds | - | | Web Application Firewall | - | | Container Threat Detection | - | | Dashboards | - | ### Dashboards Admin[​](#dashboards-admin) This role grants administrative permissions specifically for creating, managing, and sharing custom dashboards within Calico Cloud. | Feature | Permission Level | | --------------------------------- | ---------------- | | Service Graph and Flow Visualizer | - | | Policies | - | | Nodes and Endpoints | - | | Network Sets | - | | Managed Clusters | - | | Compliance Reports | - | | Timeline | - | | Alerts | - | | Kibana | - | | Image Assurance | - | | Manage Team | - | | Usage Metrics | - | | Threat Feeds | - | | Web Application Firewall | - | | Container Threat Detection | - | | Dashboards | view, edit | ## Add your own identity provider[​](#add-your-own-identity-provider) Calico Cloud works with any identity provider that supports [OpenID Connect](https://openid.net/connect/). For example, OKTA, Google, and Azure AD. To add an identity provider, open a [Support ticket](https://support.tigera.io/). ### Azure AD requirements[​](#azure-ad-requirements) To add Azure AD as your identity provider, create an Active Directory "App Registration" with a Redirect URI of type "Web" set to [https://auth.calicocloud.io/login/callback](https://auth.calicocloud.io/login/callback). Enable "ID Token" for implicit flows. Add the following Microsoft Graph API delegated permissions: - User.Read - OpenId permissions: - email - openid - profile ### Creating and assigning custom roles As an administrator, you can create custom, cluster-specific roles that restrict users to particular functions on a cluster. ## Overview[​](#overview) Calico Cloud comes with a set of predefined global roles that let you give permissions to users based on what they need to do. For example, a user with the Security global role has a broader set of permissions than a user with the Viewer global role. These permissions apply to all clusters. But in some cases these global roles can be too broad. By creating and assigning custom roles, you can be much more discriminating about what permissions you give users. For example, you could create a role that allows the user to modify network policy for a particular tier and namespace and gives view access to all other network policies. Or you can add permissions for a user to view dashboards that show data from only one namespace in a cluster. Permissions are assigned on a cluster-by-cluster basis. ## Required permissions for common Calico Cloud features[​](#required-permissions-for-common-calico-cloud-features) Certain permissions are required for a user to access common Calico Cloud features. | Feature area | Required permissions | Notes | | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Alerts | • **View Alerts** and •**View All Logs** | | | Compliance reports | • **View Compliance Reports** | | | Dashboard | • **View All Logs** and • **View Global Network Sets** or **View Network Sets** and (optional) • **View Compliance Reports** | These permissions are required for the dashboard to fully populate. All users are granted limited dashboard metrics by having access to a cluster. Users with **View Network Sets** have access to dashboards that show data only from namespaces that they have been explicitly given access to. | | Network policies | • **View** or **Modify Policies** or • **View** or **Modify Global Policies** and (optional) • **View Audit Logs** or **View All Logs** | The **Policies** permissions apply to one or more namespaces. The **Global Policies** permissions apply to the whole cluster. These permissions are also scoped by [policy tier](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/tiered-policy). The optional **View Audit Logs** or **View All Logs** let users view the change history on the policies. | | Service graph | • **View All Logs** and • **View** or **Modify Network Sets** and (optional) • **View** or **Modify Packet Captures** | Network sets can be restricted to a namespace or set to all namespaces to see all flows. | | Threat feeds | • **View** or **Modify Threat Feeds** | | | Timeline | • **View Event Logs** or **View All Logs** | | ## Before you begin[​](#before-you-begin) - You are signed in with owner or administrator permissions to the web console. ## Create a custom role and add permissions[​](#create-a-custom-role-and-add-permissions) 1. Click the user icon > **Manage Team**. 2. Under the **Roles** tab, click **Add Role**, enter a name and description for the custom role, and then click **Save**. 3. Select the cluster you want to want the role to apply to by clicking **Cluster:** and choosing the cluster. 4. Locate your new role in the list, select **Action** > **Manage permissions** > **Edit**, and the click **Add Permission**. 5. Under **Permission**, choose a permission type from the list. Depending on the permission, you may also need to choose a namespace or policy tier. 6. (optional) Click **Add permission** to add more permissions to your role for this cluster. 7. Click **Save** to save these permissions to the role for this cluster. 8. (optional) If you want to add the permissions for another cluster, repeat steps 3 to 7 for the cluster. ## Assign custom roles to a user[​](#assign-custom-roles-to-a-user) 1. Select the user icon > **Manage Team**. 2. Under the **Users** tab, locate the user in the list and select **Action** > **Edit**. 3. Select the checkboxes for each custom role you want to assign to this user and then click **Save**. ## Export custom roles and apply to other managed clusters[​](#export-custom-roles-and-apply-to-other-managed-clusters) You can export custom roles from one cluster and apply them to another cluster. > **SECONDARY:** Importing custom roles is fully supported on managed clusters running Calico Cloud 18.3 or higher. ***Prerequisites*** - You connected two or more managed clusters to Calico Cloud. - You have a managed cluster with one or more custom roles. - You have `kubectl` administrator permissions for the managed clusters you want to apply custom roles to. ***Procedure*** 1. From the cluster menu in the web console, select the managed cluster that has the custom roles you want to export. 2. Click the user icon > **Manage Team**. 3. Under the **Roles** tab, click **Export Custom Roles** and select **Download YAML** to download the custom role definitions. This file contains definitions for all the custom roles you created in this cluster. 4. For each managed cluster you want to apply the custom roles to, run the following command: ```bash kubectl apply -f roles..yaml ``` The custom roles are available immediately for this cluster in the web console. ### Give role-based access to an Entra ID group If you have Microsoft Entra ID configured as your identity provider, you can define role-based access in Calico Cloud and assign roles to an Entra ID (formerly Azure AD) security group. By managing membership in that security group, you can manage role-based access to Calico Cloud directly from your identity provider portal. ## Prerequisites[​](#prerequisites) - You are signed in to the web console with owner or administrator permissions. - Entra ID is set up as an identity provider for Calico Cloud. To set up an identity provider for Calico Cloud, open a [support ticket](https://support.tigera.io). - You have administrator permissions for your organization in the Azure Portal. - You have the Object ID for an Entra ID security group. - The **Email** property for all users in the security group has a valid email address. ## Add an Entra ID security group and assign role-based permissions for Calico Cloud[​](#add-an-entraid-security-group-and-assign-role-based-permissions-for-calico-cloud) 1. In the web console, click the user icon > **Manage Team**. 2. Select the **Roles** tab, click the **Add IdP Group** button, and then enter the following information: - **IdP Group Name**: Enter a name for the group that will be displayed in the web console. - **IdP Group Identifier**: Enter the Object ID for the Entra ID security group. - (Optional) **Description**: Enter a description for the group. > **SECONDARY:** If you don't see an **Add IdP Group** button, your organization does not have an identity provider configured. To set up an identity provider for Calico Cloud, open a [support ticket](https://support.tigera.io). 3. Select one or more predefined user roles from the **Predefined Roles** list to assign to this group. 4. Click **Save**. ## Add custom permissions to an IdP group[​](#add-custom-permissions-to-an-idp-group) 1. In the web console, click the user icon > **Manage Team**. 2. Select the **Roles** tab, and then locate the role you created for the Entra ID group. 3. Click **Actions > Manage permissions**. 4. To add permissions, locate your new IdP group role under the **Roles** tab, select **Action > Manage permissions > Edit**, and then click **Add Permission**. 5. Under **Permission**, choose a permission type from the list. Depending on the permission, you may also need to choose a namespace or policy tier. 6. Optional: To add more permissions to your role, click **Add permission** again and repeat the previous step. 7. Click **Save** to save these permissions to the role for this cluster. --- ## Tutorials ### Tutorials Learn more about Calico Cloud and Kubernetes network policy. ## Calico Cloud features[​](#calico-cloud-features) ##### [Web console tutorial](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/tour) [A quick tour of the Calico Cloud user interface.](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/tour) ##### [Service Graph tutorial](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/service-graph) [Learn the basics of using Service Graph.](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/service-graph) ##### [Understanding network sets](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/networksets) [Learn the power of network sets and why you should create them.](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/networksets) ## Secure ingress and egress for applications[​](#secure-ingress-and-egress-for-applications) ##### [Secure egress access from workloads to destinations outside the cluster](https://docs.tigera.io/calico-cloud/tutorials/applications/egress-controls) [Learn egress access controls using domains and IP addresses.](https://docs.tigera.io/calico-cloud/tutorials/applications/egress-controls) ##### [Secure ingress access to a microservice or application](https://docs.tigera.io/calico-cloud/tutorials/applications/ingress-microservices) [Create policy to secure ingress access to your microservice or application.](https://docs.tigera.io/calico-cloud/tutorials/applications/ingress-microservices) ## Implement enterprise security controls[​](#implement-enterprise-security-controls) ##### [Namespace isolation and access controls](https://docs.tigera.io/calico-cloud/tutorials/enterprise-security/namespace-isolation) [Learn how to isolate namespaces for ingress traffic.](https://docs.tigera.io/calico-cloud/tutorials/enterprise-security/namespace-isolation) ##### [Global egress access controls](https://docs.tigera.io/calico-cloud/tutorials/enterprise-security/global-egress) [Implement global egress access controls.](https://docs.tigera.io/calico-cloud/tutorials/enterprise-security/global-egress) ##### [Global default deny policy](https://docs.tigera.io/calico-cloud/tutorials/enterprise-security/default-deny) [Implement a global default deny policy in the default tier to block unwanted traffic.](https://docs.tigera.io/calico-cloud/tutorials/enterprise-security/default-deny) ##### [Platform application access controls](https://docs.tigera.io/calico-cloud/tutorials/enterprise-security/platform) [Implement ingress and egress access controls for platform applications.](https://docs.tigera.io/calico-cloud/tutorials/enterprise-security/platform) ## Kubernetes networking for beginners[​](#kubernetes-networking-for-beginners) ##### [What is network policy?](https://docs.tigera.io/calico-cloud/tutorials/training/about-network-policy) [Learn the basics of Kubernetes and Calico Cloud network policy](https://docs.tigera.io/calico-cloud/tutorials/training/about-network-policy) ##### [Kubernetes services](https://docs.tigera.io/calico-cloud/tutorials/training/about-kubernetes-services) [Learn the three main service types and how to use them.](https://docs.tigera.io/calico-cloud/tutorials/training/about-kubernetes-services) ##### [Kubernetes ingress](https://docs.tigera.io/calico-cloud/tutorials/training/about-kubernetes-ingress) [Learn the different ingress implementations and how ingress and policy interact.](https://docs.tigera.io/calico-cloud/tutorials/training/about-kubernetes-ingress) ##### [Kubernetes egress](https://docs.tigera.io/calico-cloud/tutorials/training/about-kubernetes-egress) [Learn why you should restrict egress traffic and how to do it.](https://docs.tigera.io/calico-cloud/tutorials/training/about-kubernetes-egress) ## Kubernetes tutorials and demos[​](#kubernetes-tutorials-and-demos) ##### [Kubernetes policy, demo](https://docs.tigera.io/calico-cloud/tutorials/kubernetes-tutorials/kubernetes-demo) [An interactive demo that visually shows how applying Kubernetes policy allows and denies connections.](https://docs.tigera.io/calico-cloud/tutorials/kubernetes-tutorials/kubernetes-demo) ##### [Get started with Kubernetes network policy](https://docs.tigera.io/calico-cloud/tutorials/kubernetes-tutorials/kubernetes-network-policy) [Learn Kubernetes policy syntax, rules, and features for controlling network traffic.](https://docs.tigera.io/calico-cloud/tutorials/kubernetes-tutorials/kubernetes-network-policy) ##### [Kubernetes policy, basic tutorial](https://docs.tigera.io/calico-cloud/tutorials/kubernetes-tutorials/kubernetes-policy-basic) [Learn how to use basic Kubernetes network policy to securely restrict traffic to/from pods.](https://docs.tigera.io/calico-cloud/tutorials/kubernetes-tutorials/kubernetes-policy-basic) ##### [Kubernetes policy, advanced tutorial](https://docs.tigera.io/calico-cloud/tutorials/kubernetes-tutorials/kubernetes-policy-advanced) [Learn how to create more advanced Kubernetes network policies (namespace, allow and deny all ingress and egress).](https://docs.tigera.io/calico-cloud/tutorials/kubernetes-tutorials/kubernetes-policy-advanced) ### Web console features ## [📄️ Web console tutorial](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/tour) [A quick tour of the Calico Cloud user interface.](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/tour) ## [📄️ Service Graph tutorial](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/service-graph) [Learn the basics of using Service Graph.](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/service-graph) ## [📄️ Understanding network sets](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/networksets) [Learn the power of network sets and why you should create them.](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/networksets) ## [📄️ Projects](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/projects) [About projects](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/projects) ### Web console tutorial ## What you will learn[​](#what-you-will-learn) - Web console features and controls - How to gain visibility into clusters Let's go through each item in the web console left navbar from top to bottom. You can follow along using any cluster. ## Dashboards[​](#dashboards) > From the left navbar, click Dashboards. Dashboards help you see and understand the activity in your cluster. Each dashboard is made up of categorized graphs, charts, and diagrams that visually represent the data in your logs. ![dashboards](https://docs.tigera.io/assets/images/dashboards-1379b9f25f778cc843aad5e4d93b6e2d.png) - The **Cluster Health** dashboard provides a birds-eye view of cluster activity. - The **Traffic Volume** dashboard provides a high-level view of traffic in your cluster. - The **DNS Logs** dashboard summarizes DNS data and logs into metrics, providing high-level information on the types of DNS lookups made, responses, and overall DNS performance. - The **Flow Logs** dashboard gives you an overview of how packets are being sent and received by all the pods in your cluster. - The **HTTP Traffic** dashboard provides application performance metrics for inscope Kubernetes services. - The **Network Performance** dashboard provides TCP metrics to help you identify bottlenecks, packet loss, and performance issues. ## Service Graph[​](#service-graph) > From the left navbar, select **Service Graph**, **Default** Service Graph provides a point-to-point, topographical representation of network traffic within your cluster. It is the primary tool for visibility and troubleshooting. ![service-graph](https://docs.tigera.io/assets/images/service-graph-c9f93fd4af2aca81b0f206bc33b52df1.png) **Namespaces** Namespaces are the default view in Service Graph. When you expand the top right panel `<<`, you see a detailed view of the service-to-service communications for the namespace. ![service-graph-namespace](https://docs.tigera.io/assets/images/service-graph-namespace-84290605685f69e4cbf9978f9c139c79.png) **Nodes and edges** Lines going to/from nodes are called edges. When you click on a node or edge, the right panel shows details, and the associated flow logs are automatically filtered in the bottom panel. ![edges](https://docs.tigera.io/assets/images/edges-09ed813a34ae40daaeb95a45eaca32f1.png) **Layers** Layers allow you to create meaningful groupings of resources so you can easily hide and show them on the graph. For example, you can group resources for different platform infrastructure types in your cluster like networking, storage, and logging. > Click the panel on the left (`>>`) by the Namespaces breadcrumb, and then expand the Tigera components layer. ![service-graph-layers](https://docs.tigera.io/assets/images/service-graph-layers-a00a600903ced24c407d6759bb3fd4a5.png) The **Tigera components** layer contains namespaces for Calico Cloud networking components, and a view of interest to Dev/Ops. > Click the vertical ellipses and select, **Hide layer**. Notice that only the business application namespaces remain visible in the graph. > To make this layer less visible, select **Restore layer**and click **De-emphasize layer**. **Logs, alerts, and capture jobs** The panel at the bottom below the graph provides tools for troubleshooting connectivity and performance issues. **Logs** (Flows, DNS, and HTTP) are the foundation of security and observability in Calico Cloud. When you select a node or edge in the graph, logs are filtered for the node or service. For example, here is a flow log with details including how the policies were processed in tiers. ![service-graph-flows](https://docs.tigera.io/assets/images/service-graph-flows-c6dc0245f9ec1c658e22ef37b4fc3a11.png) **Alerts** For convenience, the Alerts tab duplicates the alerts you have enabled in the **Alerts tab** in the left navbar. By default, alerts are not enabled. **Capture jobs** Service Graph integrates a packet feature for capturing traffic for a specific namespace, service, replica set, daemonset, statefulset, or pod. You can then download capture files to your favorite visualization tool like WireShark. > Right-click on any endpoint to start or schedule a capture. ![packet-capture-service](https://docs.tigera.io/assets/images/packet-capture-service-63418f90801d2a2a907bcea44ea486ae.png) **Flow Visualizations** > From the left navbar, select **Service Graph**, **Flow Visualizations**. Flow Visualizer (also called, "FlowViz") is a Calico Cloud tool for drilling down into network traffic within the cluster to troubleshoot issues. The most common use of Flow Visualizer is to drill down and pinpoint which policies are allowing and denying traffic between services. ![flow-viz](https://docs.tigera.io/assets/images/flow-viz-27724211909480cfff53f8f24bfb5b3e.png) ## Policies[​](#policies) > From the left navbar, click **Policies**. Network policy is the primary tool for securing a Kubernetes network. Policy is used to restrict network traffic (egress and ingress) in your cluster so only the traffic that you want to flow is allowed. Calico Cloud supports these policies: - Calico Cloud network policy - Calico Cloud global network policy - Kubernetes policy Calico Cloud uses **tiers** (also called, hierarchical tiers) to provide guardrails for managing network policy across teams. Policy tiers allow users with more authority (for example, Dev/ops user) to enforce network policies that take precedence over teams (for example, service owners and developers). **Policies Board** is the default view for managing tiered policies. ![policy-board](https://docs.tigera.io/assets/images/policy-export-4ad7d802ef2607aa0c8c94c4ebf2b65b.png) Users typically use a mix of Policy Board and YAML files. Note that you can export one or all policies in a tier to YAML. The **Policy Board filter** lets you filter by name, namespace, policy types and label selectors. ![policy-filters](https://docs.tigera.io/assets/images/policy-search-49291425438596bf9f9ab8f48f2d2869.png) The following features provide more security and guardrails for teams. **Recommended a policy** > In Policies Board, click **Recommend a policy**. One of the first things you'll want to do after installation is to secure unprotected pods/workloads with network policy. (For example, Kubernetes pods allow traffic from any source by default.) The Recommend a policy feature generates policies that protect specific endpoints in the cluster. Users with minimal experience with network policy can easily get started. ![recommend-policy](https://docs.tigera.io/assets/images/recommend-policy-61392143274f8323a197cc58822d1515.png) **Policy stage** When you create a policy, it is a best practice to stage it to evaluate the effects before enforcing it. After you verify that a staged network policy is allowing traffic as expected, you can enforce it. ![stage-policy](https://docs.tigera.io/assets/images/stage-policy-c28c75e6d7b7383f4c97521d25b39ea2.png) **Preview** When you edit a policy, you can select **Preview** to see how changes may affect existing traffic. ![policy-preview](https://docs.tigera.io/assets/images/policy-preview-b2efca933be175d85ece43ac331086e2.png) ## Endpoints[​](#endpoints) > From the left navbar, click **Endpoints**. **Endpoint Details** This page is a list of all pods in the cluster (also known as workload endpoints). ![endpoints](https://docs.tigera.io/assets/images/endpoint-c61da8f69c506b8d3f1ea9e07ae3a86a.png) **Node List** This page lists all nodes associated with your cluster. ![node-list](https://docs.tigera.io/assets/images/node-list-36469f7b1d899eab62352f6b9e535142.png) ## Network Sets[​](#network-sets) Network sets and global network sets are Calico Cloud resources for defining IP subnetworks/CIDRs, which can be matched by standard label selectors in policy (Kubernetes or Calico Cloud). They are a powerful feature for use/reuse and scaling policy. A simple use case is to limit traffic to/from external networks. For example, you can create a global network set with "deny-list CIDR ranges 192.0.2.55/32 and 203.0.113.0/24", and then reference the network set in a global network policy. This also allows you to see this traffic in Service Graph. ![networksets](https://docs.tigera.io/assets/images/networksets-2d9a5b9fc54f066e0d56759466069563.png) ## Managed clusters[​](#managed-clusters) > From the left navbar, click **Managed clusters**. This page is where you switch views between clusters in the web console. When you connect to a different cluster, the entire Manager view changes to reflect the selected cluster. ![managed-clusters](https://docs.tigera.io/assets/images/managed-clusters-6de294f55b709c8f94c39c821beea02a.png) ## Compliance Reports[​](#compliance-reports) > From the left navbar, click **Compliance**. Compliance tools that rely on periodic snapshots, do not provide accurate assessments of Kubernetes workloads against your compliance standards. Calico Cloud compliance dashboard and reports provide a complete inventory of regulated workloads, along with evidence of enforcement of network controls for these workloads. Additionally, audit reports are available to see changes to any network security controls. **Compliance reports** are based on archived flow logs and audit logs for all Calico Cloud resources, and audit logs for Kubernetes resources in the Kubernetes API server. ![cis-benchmark](https://docs.tigera.io/assets/images/cis-benchmark-758732e64f5f030d9be6614ded180d7b.png) Using the filter, you can select report types. ![compliance-filter](https://docs.tigera.io/assets/images/compliance-filter-5c608aa64ba8279c89e6aec62662c310.png) ## Activity[​](#activity) > From the left navbar, select **Activity**, **Timeline**. **Timeline** What changed, who did it, and when? This information is critical for security. Native Kubernetes doesn’t provide an easy way to capture audit logs for pods, namespaces, service accounts, network policies, and endpoints. The Calico Cloud timeline provides audit logs for all changes to network policy and other resources associated with your Calico Cloud deployment. ![timeline](https://docs.tigera.io/assets/images/timeline-edf282abcdc13cdb589632bcc6524495.png) > From the left navbar, selection **Activity**, **Alerts**. **Alerts** How do you know if you have an infected workload? A possible threat? Calico Cloud detects and alerts on unexpected network behavior that may indicate a security breach. You can create alerts for: - Known attacks and exploits (for example, exploits found at Shopify, Tesla, Atlassian) - DOS attempts - Attempted connections to botnets and command and control servers ![alerts](https://docs.tigera.io/assets/images/alerts-eb83344c5ce74227aab9893340fae546.png) ## Logs[​](#logs) Calico Cloud includes a fully-integrated deployment of Elastic to collect flow log data that drives key features like the Flow Visualizer, metrics in the Dashboard and Policy Board, policy automation, and testing features and security. Calico Cloud also embeds Kibana so you can view raw log data for the traffic within your cluster. > From the left navbar, click **Logs**. **Dashboards** Calico Cloud comes with built-in dashboards. ![kibana-dashboards](https://docs.tigera.io/assets/images/kibana-dashboards-0309072276c2176b09e499c94c4b1b54.png) **Log data** Kibana provides its own set of filtering capabilities to drill down into log data. For example, use filters to drill into flow log data for specific namespaces and pods. Or view details and metadata for a single flow log entry. ![kibana](https://docs.tigera.io/assets/images/kibana-b3ceea107837e68d63fc329f9e43b302.png) ## Threat feeds[​](#threat-feeds) You can add threat intelligence feeds to Calico Cloud to trace network flows of suspicious IP addresses and domains. Then, you can use network policy to block pods from contacting IPs or domains. Now that you understand the basics, we recommend the following: - [Service Graph tutorial](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/service-graph) - [Understanding policy tiers](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/tiered-policy) - [Understanding network sets](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/networksets) ### Service Graph tutorial > **SECONDARY:** This tutorial references the labs cluster setup that is part of the Calico Cloud Trial environment. ## What you will learn[​](#what-you-will-learn) - The basics of Service Graph - To visualize workload communication in the labs cluster using Service Graph - Features of Service Graph that help you manage the scale of clusters with a large number of namespaces ## The what and why of Service Graph[​](#the-what-and-why-of-service-graph) One thing we consistently hear from DevOps engineers, SREs, and platform operators is that they struggle with getting basic visibility within their Kubernetes cluster. As more apps and microservices are deployed, and workloads are changing continuously, it becomes hard to understand how everything is working together in such a complex multi-tenant environment. Service Graph provides a "weather map" view of everything in a cluster, helping new team members ramp up quickly on how everything is communicating with each other, and an easy way to engage the right stakeholders when issues come up. Service Graph delivers a point-to-point, topographical view of how namespaces, services, and deployments are communicating with each other. ## Quick demo[​](#quick-demo) **12-minute video** ## Exploring Service Graph with the labs cluster[​](#exploring-service-graph-with-the-labs-cluster) > Select the **tigera-labs cluster** in the upper right corner of the web console. In the left navigation menu, select **Service Graph**, **Default**. The default view in Service Graph provides a representation of how namespaces within the cluster are communicating with each other. Each namespace is represented with the standard Kubernetes icon for namespaces. There are some demo namespaces - **hipstershop**, **storefront**, and **acme** - and other Tigera namespaces that are used for Calico Cloud. Egress traffic leaving the cluster is represented with globe icons. Under the hood, each of these globe icons is a custom resource called a **NetworkSet**. ### Network sets[​](#network-sets) By default, Calico Cloud provides two system-defined network sets - one for public IPs called **public network** and another for private IPs called **private network**. You can also define your own network sets with a list of IPs or CIDRs and they will appear on this graph, enabling you to get more granular visibility into the egress traffic that is specific to your environment. The **public-ip-range** is an example of such a network set that has been already created in the labs cluster. For more examples of using network sets, see [Secure egress access from workloads to destinations outside the cluster](https://docs.tigera.io/calico-cloud/tutorials/applications/egress-controls), and [Understanding network sets](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/networksets). ### Selecting graph edges and graph nodes[​](#selecting-graph-edges-and-graph-nodes) > Click the `<<` tab in the upper right to open the details panel. The details panel provides additional information on a selected graph node or edge. The log panel below the graph also automatically updates and/or filters data (if available) for the selection. > Click on the edge between the **hipstershop** namespace, and the network set **public network**. The flow logs are automatically filtered below in the Flows tab, and you can expand any of the rows in that tab to view some of the detailed metadata that Calico Cloud provides around workload communication. ### Double-clicking on a namespace[​](#double-clicking-on-a-namespace) You can also double-click on a namespace to see services and deployments within a namespace, and how they are communicating externally, and with the rest of the cluster. > Double-click **hipstershop** or **storefront**. All of the resources in purple are part of the selected namespace (which is also listed in the breadcrumb in the upper left), and anything external to the namespace is represented in blue. These views are useful when troubleshooting a specific application or microservice running in your cluster. > To return the default view, click the **Namespaces** breadcrumb in the upper left. ### Right-click actions to manage scale[​](#right-click-actions-to-manage-scale) Right-clicking a namespace, service, or deployment brings up another set of actions that are designed to help you manage the scale of your Kubernetes cluster. Although the labs cluster has only a dozen or so namespaces, your Kubernetes clusters could likely have over one hundred namespaces. > Right-click **hipstershop**, and select **Hide unrelated**. Hide unrelated allows you to quickly filter and trim the Service Graph to show just the selected entity and anything it is communicating with. This is helpful in troubleshooting issues for a specific namespace or application, and for application teams to quickly understand their upstream and downstream dependencies. > To reset the view after this action, select the **Reset icon** (vertical column of icons), and click, **Reset view & filters**. ## Using layers and views[​](#using-layers-and-views) Another unique feature of Service Graph is the concept of **layers**, which allows you to create meaningful groupings of resources so you can easily hide and show them on the graph. It is another tool to help you manage the overall scale of visualizing workloads within your cluster. You can create layers for different types of platform infrastructure you might have in your cluster - networking, storage, logging. > Click the `>>` tab in the upper left (next to Namespaces) to open the panel. Expand the Tigera components layer to view its namespaces. The layer called **Tigera components** contains namespaces related to Calico Cloud. Click the ellipsis/spillover menu on the right, and select **Hide layer**. This hides the entire layer on the graph, making it easy to hide/show a group of related namespaces. > Reset the view by selecting, **Restore layer**. ### Views[​](#views) **Views** allow you to save the state of the graph on the canvas. > Click the **Views** (panel above Layers). Let's create a view that shows only items related to the hipstershop product catalog. > Double-click **hipstershop**, select the *productcatalogservice*, right-click and select **Hide unrelated to service group**. In the View tab, click **Save Current** and save the view as "Product Catalog”. > Click the **Namespaces** breadcrumb to reset the views and filters. In the Views tab, you can see your saved view, "Product Catalog" -- so you can quickly revisit this view at any time. ## Logs and details panel[​](#logs-and-details-panel) As you are working with Service Graph, the bottom panel and the details panel (right of the main canvas) will display additional information for selections you make on the graph. The exact information displayed will vary depending on the following: - Whether you select a graph node or edge - Type of graph node selected (namespace, network set, service, etc.) - Whether or not you have deployed Envoy for Layer 7 visibility ### Details panel[​](#details-panel) > Double-click the **hipstershop** namespace to see a detailed view of service-to-service communication. Select the **checkoutservice** on the graph. You can see the **Inbound** and **Outbound** connections for this service. Expanding any of those connections (hovering over the line and clicking the arrow) shows you volumetric data for each of those flows. In the **Insights** section, you can click on items like DNS stats in use for the service. > In the line, `checkoutservice >> shippingservice`, click the green arrows to display protocols and policies. The **Process Info** section shows you specific processes associated with the traffic flows for the `checkoutservice`. This can be helpful in troubleshooting scenarios where identifying the specific binaries that are involved in service communication may be required to track down a bug. ### Logs panel[​](#logs-panel) In the bottom panel are several tabs: Flows, DNS, HTTP, Alerts, and Capture Jobs. With the `checkoutservice` still selected, filters are automatically applied on the Flows tab with detailed logs for Layer 4 traffic. Similar filters are automatically applied to the DNS and HTTP flows. Note that HTTP flows require the deployment of Envoy to see Layer 7 traffic in any cluster connected to Calico Cloud. The Alerts tab will show alerts if they are generated, and Capture Jobs will show any packet capture jobs that have been defined for this namespace. Now that you understand the basics for Service Graph, we recommend: - [Understanding policy tiers](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/tiered-policy) - [Understanding network sets](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/networksets) ### Understanding network sets ## Visualize traffic to/from your cluster[​](#visualize-traffic-tofrom-your-cluster) Modern applications often integrate with third-party APIs and SaaS services that live outside Kubernetes clusters. To securely enable access to those integrations, you must be able to limit IP ranges for egress and ingress traffic to workloads. Limiting IP lists or ranges is also used to deny-list bad actors or embargoed countries. To limit IP ranges, you need to use the Calico Cloud resource called **network sets**. ## What are network sets?[​](#what-are-network-sets) **Network sets** are a grouping mechanism that allows you to create an arbitrary set of IP subnetworks/CIDRs or domains that can be matched by standard label selectors in Kubernetes or Calico Cloud network policy. Like IP pools for pods, they allow you to reuse/scale sets of IP addresses in policies. A **network set** is a namespaced resource that you can use with Kubernetes or Calico Cloud network policies; a **global network set** is a cluster-wide resource that you can use with Calico Cloud network policies. Like network policy, you manage user access to network sets using standard Kubernetes RBAC. ## Why are network sets powerful?[​](#why-are-network-sets-powerful) If you are familiar with Service Graph in the web console, you know the value of seeing pod-to-pod traffic within your cluster. But what about traffic external to your cluster? Calico Cloud automatically detects IPs for pods and nodes that fall into the standard IETF “public network” and “private network” designations, and displays those as icons in Service Graph. So you get some visibility into external traffic without using any network sets. ![public-private-networks](https://docs.tigera.io/assets/images/public-private-networks-58c5e15f90f09422c523d29fff1fc73b.png) However, when you create network sets, you can get more granular visibility into what's leaving the cluster to public networks. Because you control the grouping, the naming, and labeling, you create visibility that is customized to your organization. This is why they are so powerful. Here are just a few examples of how network sets can be used: - **Egress access control** Network sets are a key resource for defining egress access controls; for example, securing ingress to microservices/apps or egress from workloads outside the cluster. - **Troubleshooting** Network sets appear as additional metadata in flow logs and Kibana, Flow Visualizer, and Service Graph. - **Efficiency and scaling** Network sets are critical when scaling your deployment. You may have only a few CIDRs when you start. But as you scale out, it is easier to update a handful of network sets than update each network policy individually. Also, in a Kubernetes deployment, putting lots of anything (CIDRs, ports, policy rules) directly into policies causes inefficiencies in traffic processing (iptables/eBPF). - **Microsegmentation and shift left** Network sets provide the same microsegmentation controls as network policy. For example, you can allow specific users to create policies (that reference network sets), but allow only certain users to manage network sets. - **Threat defense** Network sets are key to being able to manage threats by blocking bad IPs with policy in a timely way. Imagine having to update individual policies when you find a bad IP you need to quickly block. You can even give access to a controller that automatically updates CIDRs in a network set when a bad IP is found. ## Create a network set and use it in policy[​](#create-a-network-set-and-use-it-in-policy) In this section, we’ll walk through how to create a namespaced network set in the web console. You can follow along using your cluster or tigera-labs cluster. In this example, you will create a network set named, `google`. This network set contains a list of trusted google endpoints for a microservice called, `hipstershop`. As a service owner, you want to be able to see traffic leaving the microservices in Service Graph. Instead of matching endpoints on IP addresses, we will use domain names. 1. From the left navbar, click **Network Sets**. 2. Click **Add Network Set**, and enter these values. - For Name: `google` - For Scope: Select **Namespace** and select, `hipstershop` 3. Under Labels, click **Add label**. - In the Select key field, enter `destinations` and click the green bar to add this new entry. - In the Value field, enter `google`, click the green bar to add the entry, and save. 4. For Domains, click **+Add Domain** and these URLs: `clouddebugger.googleapis.com`, `cloudtrace.googleapis.com`, `metadata.google.internal`, `monitoring.googleapis.com`. 5. Click **Create Network Set**. You’ve created your first network set. ![add-networkset-google](https://docs.tigera.io/assets/images/add-networkset-google-383ea2b92cdea32c79367d937e2e5dca.png) The YAML looks like this: ```yaml kind: NetworkSet apiVersion: projectcalico.org/v3 metadata: name: google labels: destinations: google namespace: hipstershop spec: nets: [] allowedEgressDomains: - clouddebugger.googleapis.com - cloudtrace.googleapis.com - metadata.google.internal - monitoring.googleapis.com ``` Next, we write a DNS policy for hipstershop that allows egress traffic to the trusted google sites. The following network policy allows egress access for all destination selectors labeled, `google`. Note that putting domains in a network set referencing it in policy is the best practice. Also, note that using `selector: all()` should only be used if all pods in the namespace can access all of the domains in the network set; if not, you should create separate policies accordingly. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: application.allow-egress-domain namespace: hipstershop spec: tier: application order: 0 selector: all() serviceAccountSelector: '' egress: - action: Allow source: {} destination: selector: destinations == "google" types: - Egress ``` ## Network sets in Service Graph[​](#network-sets-in-service-graph) Continuing with our `hipstershop` example, if you go to Service Graph, you see hipstershop (highlighted in yellow). ![hipstershop](https://docs.tigera.io/assets/images/hipstershop-869531dc562c3a6c74653462edd412b4.png) If we double-click `hipstershop` to drill down, we now see the `google` network set icon (highlighted in yellow). We now have visibility to traffic external from google sites to hipstershop. (If you are using the tigera-labs cluster, note that the network set will not be displayed as shown below.) ![google-networkset](https://docs.tigera.io/assets/images/google-networkset-d155ececbd4bce3faf3e0436b479188e.png) Service Graph provides a view into how services are interconnected in a consumable view, along with easy access to flow logs. However, you can also see traffic associated with network sets in volumetric display with Flow Visualizer, and query flow log data associated with network sets in Kibana. ## Tutorial[​](#tutorial) In the following example, we create a global network set resource for a trusted load-balancer that can be used with microservices and applications. The label, `trusted-ep: load-balancer` is how this global network set can be referenced in policy. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkSet metadata: name: load-balancer labels: trusted-ep: "load-balancer" spec: nets: # Modify the ip addresses to refer to the ip addresses of load-balancers in your environment - 10.0.0.1/32 - 10.0.0.2/32 ``` The following network policy uses the `selector: trusted-ep == "load balancer"` to reference the above GlobalNetworkSet. All applications in the `app2-ns` namespace, that match `app2` and `svc1` are allowed ingress traffic from the trusted load balance on port 1001. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: application.app2-svc1 namespace: app2-ns spec: tier: application order: 500 selector: (app == "app2"&&svc == "svc1") ingress: - action: Allow protocol: TCP source: selector: trusted-ep == "load-balancer" destination: ports: - '10001' types: - Ingress ``` ### Advanced policy rules with network sets[​](#advanced-policy-rules-with-network-sets) When you combine Calico Cloud policy rules with network sets, you have powerful ways to fine-tune. The following example combines network sets with specific rules in a global network policy to deny access more quickly. We start by creating a Calico Cloud GlobalNetworkSet that specifies a list of CIDR ranges we want to deny: 192.0.2.55/32 and 203.0.113.0/24. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkSet metadata: name: ip-protect labels: ip-deny-list: 'true' spec: nets: - 192.0.2.55/32 - 203.0.113.0/24 ``` Next, we create two Calico Cloud GlobalNetworkPolicy resources. The first is a high "order" policy that allows traffic as a default for things that don’t match our second policy, which is low "order" and uses the GlobalNetworkSet label as a selector to deny ingress traffic (IP-deny-list in the previous step). In the label selector, we also include the term, `!has(projectcalico.org/namespace)`, which prevents this policy from matching pods or NetworkSets that also have this label. To more quickly enforce the denial of forwarded traffic to the host at the packet level, use the `doNotTrack` and `applyOnForward` options. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: forward-default-allow spec: selector: apply-ip-protect == 'true' order: 1000 doNotTrack: true applyOnForward: true types: - Ingress ingress: - action: Allow --- apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: ip-protect spec: selector: apply-ip-protect == 'true' order: 0 doNotTrack: true applyOnForward: true types: - Ingress ingress: - action: Deny source: selector: ip-deny-list == 'true' && !has(projectcalico.org/namespace) ``` ## Best practices for using network sets[​](#best-practices-for-using-network-sets) - Create network sets as soon as possible after getting started This allows you to quickly realize the benefits of seeing custom metadata in flow logs and visualizing traffic in Service Graph and Flow Visualizer. - Create a network set label and name schema It is helpful to think: what names would be meaningful and easy to understand when you look in Service Graph? Flow Viz? Kibana? What labels will be easy to understand when used in network policies – especially if you are separating users who manage network sets from those who consume them in network policies. - Do not put large sets of CIDRs and domains directly in policy Network sets allow you to specify CIDRs and/or domains. Although you can add CIDRs and domains directly in policy, it doesn't scale. - Do not put thousands of rules into a policy, each with a different CIDR If your set of /32s can be easily aggregated into a few broader CIDRs without compromising security, it’s a good thing to do; whether you’re putting the CIDRs in the rule or using a network set. - If you want to match thousands of endpoints, write one or two rules and use selectors to match the endpoints. Having one rule per port, per host is inefficient because each rule ends up being rendered as an iptables/eBPF rule instead of making good use of IP sets. - Avoid overlapping IP addresses/subnets in networkset/globalnetworkset definitions The following table provides guidance on the efficient use of network sets. | Policy | Network set | Results | | ----------------------------------------------------------------------------------------------------- | -------------------------------------- | ----------------------------------------------------------------------------------------------------- | | source: selector: foo="bar" | With handful of broad CIDRs | **Efficient** **-** 1 iptables/eBPF rule - 1 IP set with handful of CIDRs | | source: nets: \[ ... handful ...] | Not used | **Efficient** - Handful of iptables/eBPF rules - 0 IP sets | | source: selector: foo="bar" | One network set with 2000 x /32s | **Fairly efficient** - 1 iptables/eBPF rule - 1 IP sets with 2000 entries | | | Two network sets with 1000 each x /32s | **Efficient** - 2 iptables/eBPF rules - 2 IP set with 1000 entries | | source: nets: \[... 2000 /32s ...] - source: nets: \[1 x /32] - source: nets: \[1 x /32] - ... x 2000 | Not used | **Inefficient** Results in programming 2k iptables/eBPF rules - 2000+ iptables/eBPF rules - 0 IP sets | For more examples of network sets with policy, see: - Namespaced network set - [Secure ingress access to a microservice or application](https://docs.tigera.io/calico-cloud/tutorials/applications/) - Global network set - [Secure egress access from workloads to destinations outside the cluster](https://docs.tigera.io/calico-cloud/tutorials/applications/egress-controls) - [Global egress access controls](https://docs.tigera.io/calico-cloud/tutorials/enterprise-security/global-egress) ### Projects You can use projects to organize your managed clusters into logical groupings. ![Project page](https://docs.tigera.io/img/calico-cloud/calico-cloud-projects.png) ## About projects[​](#about-projects) In Calico Cloud, a *project* is a logical grouping of one or more managed clusters. You can use projects to organize your managed clusters in ways that are useful for your organization. For example, you can use projects to segregate clusters by product teams, or by geo location. When you view observability tools in Calico Cloud, you see only what is available for the current project. All Calico Cloud organizations start with a single default project. You can have up to three projects at a given time. ## Create a project[​](#create-a-project) **Prerequisites** - You have Admin or Owner privileges. **Procedure** 1. From any page in the web console, click the **Project** menu and then click **Manage Projects**. 2. On the **Projects** page, click **Add Project**. 3. Enter a name and description for the project, and then click the **Add Project** button. Provisioning the new project takes a few minutes. When provisioning is complete, you can begin connecting clusters. ## Connect a cluster to a project[​](#connect-a-cluster-to-a-project) - To connect a cluster to a specific project, make sure you have selected the project you want as you follow the procedure to [connect your cluster to Calico Cloud](https://docs.tigera.io/calico-cloud/get-started/install-cluster). ## Move a managed cluster to another project[​](#move-a-managed-cluster-to-another-project) You can move a managed cluster to another project by removing it from the original project and connecting it to the new project. > **WARNING:** Log data held by Calico Cloud will be deleted when you remove your cluster from the original project. When you reconnect your cluster to Calico Cloud as part of the new project, flow logs will start to be collected again. You will not be able to restore access to the old logs. > > If you need to retain logs, contact [our support team](https://tigeraio.my.site.com/community/s/login/). 1. To remove your cluster from Calico Cloud, find your cluster on the **Managed Clusters** page, and then click **Actions > Delete**. After a moment, the cluster is removed from the list. 2. Move to the new project by clicking the **Projects** menu and selecting your project. 3. To reconnect your cluster, follow the guide to [connect your cluster to Calico Cloud](https://docs.tigera.io/calico-cloud/get-started/install-cluster). ### Secure ingress and egress for applications ## [📄️ Secure ingress access to a microservice or application](https://docs.tigera.io/calico-cloud/tutorials/applications/ingress-microservices) [Create policy to secure ingress access to your microservice or application.](https://docs.tigera.io/calico-cloud/tutorials/applications/ingress-microservices) ## [📄️ Secure egress access from workloads to destinations outside the cluster](https://docs.tigera.io/calico-cloud/tutorials/applications/egress-controls) [Learn egress access controls using domains and IP addresses.](https://docs.tigera.io/calico-cloud/tutorials/applications/egress-controls) ### Secure ingress access to a microservice or application In this article you will learn how to create Calico Cloud policies that securely allow ingress access to a microservice or application. **Tab: Ingress access to a microservice** ## Secure ingress access to a microservice[​](#secure-ingress-access-to-a-microservice) In this example, we have a microservices with four services (sv1, sv2, sv3, and sv4). Let's assume the following requirements: - Everyone internal to the organization can access `svc1` pods at TCP port 10001 - `svc1` pods can access `svc2` at TCP port 10002 - `svc3` pods can access at TCP port 10003 - `svc2` pods can access `vc4` at TCP ports 10004, 10005 Also, because `svc1` pods pass through a trusted load balancer, we want to allow ingress traffic for the load balancer. ![ingress-microservices](https://docs.tigera.io/assets/images/ingress-microservices-ef6286e6a51e1b25c363fee53294e1b3.png) Let's start with securing ingress access to the trusted load balancer. We will use the Calico Cloud **GlobalNetworkSet** resource with cluster-wide scope to define the load balancer IP addresses; this allows the same trusted external endpoints to be used across multiple namespaces. **GlobalNetworkSet for load balancer** This GlobalNetworkSet contains the IP addresses for the trusted load balancer. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkSet metadata: name: load-balancer labels: trusted-ep: "load-balancer" spec: nets: # Modify the ip addresses to refer to the ip addresses of load-balancers in your environment - 10.0.0.1/32 - 10.0.0.2/32 ``` Next, we will create four network policies. **NetworkPolicy 1** This policy allows ingress from the trusted load balancer to pods in `sv1`. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: application.app2-svc1 namespace: app2-ns spec: tier: application order: 500 selector: (app == "app2" && svc == "svc1") ingress: - action: Allow protocol: TCP source: selector: trusted-ep == "load-balancer" destination: ports: - '10001' types: - Ingress ``` **NetworkPolicy 2** This policy allows ingress access from `sv1` pods to `sv2` pods. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: application.app2-svc2 namespace: app2-ns spec: tier: application order: 600 selector: (app == "app2" && svc == "svc2") ingress: - action: Allow protocol: TCP source: selector: svc == "svc1" destination: ports: - '10002' types: - Ingress ``` **NetworkPolicy 3** This policy allows ingress access from `svc1` pods to `svc3` pods. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: application.app2-svc3 namespace: app2-ns spec: tier: application order: 700 selector: (app == "app2" && svc == "svc3") ingress: - action: Allow protocol: TCP source: selector: svc == "svc1" destination: ports: - '10003' types: - Ingress ``` **NetworkPolicy 4** This policy allows ingress access from `svc2` pods to `svc4` pods. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: application.app2-svc4 namespace: app2-ns spec: tier: application order: 800 selector: (app == "app2" && svc == "svc4") ingress: - action: Allow protocol: TCP source: selector: svc == "svc2" destination: ports: - '10003' - '10004' types: - Ingress ``` **Tab: Ingress access to an application** ## Secure ingress access to an application[​](#secure-ingress-access-to-an-application) In this example, we have an application with frontend, backend, and a database. Let's assume the following requirements: - Everyone internal to the organization can access the `frontend` at TCP port 10001 - `frontend` can access the `backend` at TCP port 10002 - `backend` can access the `database` at TCP ports 10003, 10004 Also, because frontend pods pass through a trusted load balancer, we want to allow ingress traffic for the load balancer. ![ingress-application](https://docs.tigera.io/assets/images/ingress-application-fa4e37abc513375c7eb5f5129a06a497.png) Let's start with securing ingress access to the trusted load balancer. We will use the Calico Cloud **GlobalNetworkSet** resource with cluster-wide scope to define the load balancer IP addresses; this allows the same trusted external endpoints to be used across multiple namespaces. **GlobalNetworkSet for load balancer** This GlobalNetworkSet contains the IP addresses for the trusted load balancer. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkSet metadata: name: load-balancer labels: trusted-ep: "load-balancer" spec: nets: # Modify the ip addresses to refer to the ip addresses of load-balancers in your environment - 10.0.0.1/32 - 10.0.0.2/32 ``` Next, we will create three network policies. **NetworkPolicy 1** This policy allows ingress from the trusted load balancer to pods in `frontend`. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: application.app1-frontend namespace: app1-ns spec: tier: application order: 200 selector: (app == "app1" && svc == "frontend") serviceAccountSelector: '' ingress: - action: Allow protocol: TCP source: selector: trusted-ep == "load-balancer" destination: ports: - '10001' types: - Ingress ``` **NetworkPolicy 2** This policy allows ingress access from pods in the `frontend` to pods in the `backend`. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: application.app1-backend namespace: app1-ns spec: tier: application order: 300 selector: (app == "app1" && svc == "backend") serviceAccountSelector: '' ingress: - action: Allow protocol: TCP source: selector: svc == "frontend" destination: ports: - '10002' types: - Ingress ``` **NetworkPolicy 3** This policy allows ingress access from pods in the `backend` to pods in the `database`. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: application.app1-database namespace: app1-ns spec: tier: application order: 400 selector: (app == "app1" && svc == "database") serviceAccountSelector: '' ingress: - action: Allow protocol: TCP source: selector: svc == "backend" destination: ports: - '10003' - '10004' types: - Ingress ``` ### Secure egress access from workloads to destinations outside the cluster In this article we'll show you how to restrict egress access for your application or microservice pods to external endpoints outside of the cluster. - [Use a network set with a network policy](#use-a-network-set-with-a-network-policy) - [Use wildcards in domain names](#use-wildcards-in-domain-names) - [Use a global network set with a global network policy](#use-a-global-network-set-with-a-global-network-policy) ## What are networks sets?[​](#what-are-networks-sets) Calico Cloud NetworkSet and GlobalNetworkSet resources are used to define endpoints external to the Kubernetes cluster. The scope of a NetworkSet is the **namespace where they are defined**; the scope of a GlobalNetworkSet is **cluster-wide**. NetworkSet/GlobalNetworkSet come in two types: - **Network-based** - defines IP address-based external endpoints - **Domain-based** - defines URL-based external endpoints. Using domains in policy is often called, DNS policy. ## Use a network set with a network policy[​](#use-a-network-set-with-a-network-policy) In this example, we have a microservice that requires egress access to two external endpoints: a repo and a partner. ![egress-example](https://docs.tigera.io/assets/images/egress-example-f30548b0681bc177660e3841db548959.png) `svc3` pods needs egress access to: - A repo named, `app2-repo` at domain `app2-repo.example.com`, port 443 - A partner named, `app2-partners` at endpoint `10.10.10.10/32`, port 1010 and 53 First, we define a domain-based NetworkSet. Using `allowedEgressDomains` we can specify the trusted repo by its URL, `app2-repo.example.com`. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkSet metadata: name: app2-repo namespace: app2-ns labels: trusted-ep: 'app2-repo' spec: allowedEgressDomains: - 'app2-repo.example.com' ``` Next, we create a network-based NetworkSet that specifies the IP address of the trusted partner, `10.10.10.10/32`. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkSet metadata: name: app2-partners namespace: app2-ns labels: trusted-ep: 'app2-partners' spec: nets: - 10.10.10.10/32 ``` Now we can reference these NetworkSets by their labels in NetworkPolicy. We use a selector to specify the service, `app == "app2"&&svc == "svc3"`, and then selectors to allow egress to our two trusted endpoints, `selector: trusted-ep == "app2-repo"` at port 443, and `selector: trusted-ep == "app2-partners"` at ports 1010 and 53. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: application.app2-svc3-egress namespace: app2-ns spec: tier: application selector: (app == "app2" && svc == "svc3") egress: - action: Allow source: {} destination: selector: trusted-ep == "app2-repo" ports: - '443' protocol: TCP - action: Allow source: {} destination: selector: trusted-ep == "app2-partners" ports: - '1010' protocol: TCP - action: Allow protocol: UDP source: {} destination: ports: - '53' types: - Egress ``` ## Use wildcards in domain names[​](#use-wildcards-in-domain-names) In this example, we create another namespaced NetworkPolicy with egress rules with `action: Allow` and a `destination.domains` field specifying the domain names to which egress traffic is allowed. The first egress rule allows DNS traffic using UDP over port 53, and the second rule allows connections outside the cluster to domains `api.alice.com` and `*.example.com` (which means `.example.com`, such as `bob.example.com`). Note that our namespaced NetworkPolicy can only grant egress access to specified domains, and to workload endpoints in the `rollout-test` namespace. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: allow-egress-to-domains namespace: rollout-test spec: order: 1 selector: my-pod-label == 'my-value' types: - Egress egress: - action: Allow protocol: UDP destination: ports: - 53 - dns - action: Allow destination: domains: - api.alice.com - '*.example.com' ``` ## Use a global network set with a global network policy[​](#use-a-global-network-set-with-a-global-network-policy) We recommend using a GlobalNetworkSet when the same set of domains needs to be referenced in multiple policies, or when you want the allowed destinations to be a mix of domains and IPs from global network sets, or IPs from workload endpoints and host endpoints. By using a single destination selector in a global network set, you can potentially match all of these resources. In the following example, the allowed egress domains (`api.alice.com` and `*.example.com`) are specified in the GlobalNetworkSet. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkSet metadata: name: allowed-domains-1 labels: color: red spec: allowedEgressDomains: - api.alice.com - '*.example.com' ``` Then, we reference the global network set in a GlobalNetworkPolicy using a destination label selector. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: allow-egress-to-domain spec: order: 1 selector: my-pod-label == 'my-value' types: - Egress egress: - action: Allow destination: selector: color == 'red' ``` ## For reference...[​](#for-reference) ### Allowed egress domains[​](#allowed-egress-domains) Using domain names in policy rules is limited to only egress allow rules. Calico Cloud allows connections only to IP addresses returned from DNS lookups to trusted DNS servers. The supported DNS types are: A, AAAA, and CNAME records. The domain name must be an exact match; for example, **google.com** is treated as distinct from **[www.google.com](http://www.google.com)**. **Note:** Kubernetes labels provide a similar convenience for services within the cluster. Calico Cloud does not support using domain names for services within the cluster. Use Kubernetes labels for services within the cluster. ### Domain name matching[​](#domain-name-matching) When a configured domain name has no wildcard (`*`), it matches exactly that domain name. For example: - `microsoft.com` - `tigera.io` With a single asterisk in any part of the domain name, it matches 1 or more path components at that position. For example: - `*.google.com` matches `www.google.com` and `www.ipv6.google.com`, but not `google.com` - `www.*.com` matches `www.sun.com` and `www.apple.com`, but not `www.com` - `update.*.mycompany.com` matches `update.tools.mycompany.com`, `update.secure.suite.mycompany.com`, and so on **Not** supported are: - Multiple wildcards in the same domain, for example: `*.*.mycompany.com` - Asterisks that are not the entire component, for example: `www.g*.com` - More general wildcards, such as regular expressions ### Workload and host endpoints[​](#workload-and-host-endpoints) Policy with domain names can be enforced on workload or host endpoints. When a policy with domain names applies to a workload endpoint, it allows that workload to connect out to the specified domains. When policy with domain names applies to a host endpoint, it allows clients directly on the relevant host (including any host-networked workloads) to connect out to the specified domains. ### Trusted DNS servers[​](#trusted-dns-servers) Calico Cloud trusts DNS information only from its list of DNS trusted servers. Using trusted DNS servers to back domain names in policy, prevents a malicious workload from using IPs returned by a fake DNS server to hijack domain names in policy rules. By default, Calico Cloud trusts the Kubernetes cluster’s DNS service (kube-dns or CoreDNS). For workload endpoints, these out-of-the-box defaults work with standard Kubernetes installs, so normally you won’t change them. For host endpoints you will need to add the IP addresses that the cluster nodes use for DNS resolution. To change the default DNS trusted servers, use the DNSTrustedServers parameter in Felix. ### Implement enterprise security controls ## [📄️ Namespace isolation and access controls](https://docs.tigera.io/calico-cloud/tutorials/enterprise-security/namespace-isolation) [Learn how to isolate namespaces for ingress traffic.](https://docs.tigera.io/calico-cloud/tutorials/enterprise-security/namespace-isolation) ## [📄️ Global egress access controls](https://docs.tigera.io/calico-cloud/tutorials/enterprise-security/global-egress) [Implement global egress access controls.](https://docs.tigera.io/calico-cloud/tutorials/enterprise-security/global-egress) ## [📄️ Global default deny policy](https://docs.tigera.io/calico-cloud/tutorials/enterprise-security/default-deny) [Implement a global default deny policy in the default tier to block unwanted traffic.](https://docs.tigera.io/calico-cloud/tutorials/enterprise-security/default-deny) ## [📄️ Platform application access controls](https://docs.tigera.io/calico-cloud/tutorials/enterprise-security/platform) [Implement ingress and egress access controls for platform applications.](https://docs.tigera.io/calico-cloud/tutorials/enterprise-security/platform) ### Namespace isolation and access controls In this article you will learn how to create Calico Cloud global network policies to isolate namespaces for ingress traffic. ## Isolate access[​](#isolate-access) Isolating ingress traffic access can be implemented across different groups in your organization, for example: - Business unit, development team, or project - Deployment environment (Dev/Stage/Prod or Pre-prod/Prod) - Control plane - Management plane - Compliance environment ## Create global network policies[​](#create-global-network-policies) Microsegmentation across namespaces is achieved using Calico Cloud policy labels and selectors with a **Pass** action rule. If you are unfamiliar with policy action rules, see [Tutorial - Understanding policy tiers](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/tiered-policy). In the following example, let's assume the following: - We created a **security tier** (recommended for your enterprise security controls like multi-tenancy) - Two development teams (`dev1` and `dev2`), each managing two applications or microservices - The security tier will delegate (pass) microsegmentation controls to the application tier - We will use a GlobalNetworkPolicy with cluster wide scope that applies to pods in all namespaces in the cluster. For example: ![multi-tenancy-ingress](https://docs.tigera.io/assets/images/multi-tenancy-ingress-8cd5267a19107f50b8393fa7ff179001.png) **GlobalNetworkPolicy 1** The following policy is for ingress traffic that goes through a trusted load balancer. It resides in the security tier, and uses the **Pass** action rule to delegate control to the application tier for the `dev1` tenant. Because the application tier contains endpoints that apply, traffic is processed. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: security.ingress-dev1 spec: tier: security order: 200 selector: tenant == "dev1" ingress: - action: Pass source: selector: trusted-ep == "load-balancer" - action: Pass source: selector: tenant == "dev1" types: - Ingress ``` **GlobalNetworkPolicy 2** Similarly, we create a second GlobalNetworkPolicy for `dev2` tenant that delegates control to the application tier for the `dev2` applications. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: security.ingress-dev2 spec: tier: security order: 300 selector: tenant == "dev2" ingress: - action: Pass source: selector: trusted-ep == "load-balancer" - action: Pass source: selector: tenant == "dev2" types: - Ingress ``` ## Summary and recommendations[​](#summary-and-recommendations) In these examples, we implemented ingress access controls for two development teams. The global network policy has cluster-wide scope (applying to pods in all namespaces in the cluster). For **egress controls**, we recommend the following: - Use permissive egress policies for intra-cluster communication - Implement [Global default deny](https://docs.tigera.io/calico-cloud/tutorials/enterprise-security/default-deny) security controls - Implement [Global egress access controls](https://docs.tigera.io/calico-cloud/tutorials/enterprise-security/global-egress) for communication with endpoints external to the cluster in the security tier. These controls govern the communication of all applications for a given tenant with the outside world. ### Global egress access controls In this article you will learn how to implement egress access controls cluster-wide for all applications and microservices. In this example, we will implement global egress access controls for **dev1 team**: - Egress access control for all applications managed by dev1: applications (**app1**) and microservices (**app2**) - dev1 pods can egress access to a repo named, `repo.acme.corp` at port 443 - dev1 pods can egress access to a trusted partner (in this example, another business unit in the organization) at `10.10.10.10` on port 1010. ![global-egress](https://docs.tigera.io/assets/images/global-egress-964438f93c84374ff6fac7b603233e31.png) ## Create global network sets[​](#create-global-network-sets) First, we need to create a GlobalNetworkSet for the trusted repo, `repo.acme.corp`. Because we already have a label taxonomy following the best practices, it is easy. Just use the `allowedEgressDomains` to specify the trusted repo, `repo.acme.corp`. ```yaml kind: GlobalNetworkSet apiVersion: projectcalico.org/v3 metadata: name: trusted-repo labels: trusted-ep: dev1-repo spec: allowedEgressDomains: - repo.acme.corp ``` Next, we create a separate global network set for our trusted business unit within the organization. We use the `dev1-partners` label, and specify the IP address, `10.10.10.10/32`. ```yaml kind: GlobalNetworkSet apiVersion: projectcalico.org/v3 metadata: name: trusted-partners labels: trusted-ep: dev1-partners spec: nets: - 10.10.10.10/32 ``` ## Create global network policy[​](#create-global-network-policy) Before we create our GlobalNetworkPolicy, let's review our labels and set up. - We assume that app1 and app2 and all other dev1 team apps’ pods have a label that identifies dev1 (tenant: dev1) to enforce the declared controls. - For selected pods (all dev1 pods), all egress traffic that is not explicitly allowed is denied, as part of the best practices for policy tiers. - The policy below, which is allowing egress communication of dev1 pods with all other pods in the cluster, is done by selecting all namespaces as the destination. This means we are applying granular controls to pods for traffic destined to external endpoints, and we are allowing egress traffic destined to other pods in the cluster. This is recommended unless you have specific control requirements that dictate otherwise. This simplifies policy creation since we have already defined granular ingress controls for intra-cluster pod-to-pod communication. The alternative would be to define granular ingress and egress controls to all pods, which adds complexity in policy development. Here is our GlobalNetworkPolicy. The dev1 team is allowed egress access to the trusted repo, `dev-repo` via port 443, and also to the trusted business unit, `dev-partners` via port 1010. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: security.egress-dev1 spec: tier: security selector: tenant == "dev1" egress: - action: Allow protocol: TCP source: {} destination: selector: trusted-ep == "dev1-repo" ports: - '443' - action: Allow protocol: TCP source: {} destination: selector: trusted-ep == "dev1-partners" ports: - '1010' - action: Allow source: {} destination: namespaceSelector: all() types: - Egress ``` As you can see, once you have your labels in place, creating policy to secure teams is straightforward. ### Global default deny policy In this article you will learn when and how to create a global default deny policy for the cluster. ## What is it, when should you create one?[​](#what-is-it-when-should-you-create-one) A global default deny policy ensures that unwanted traffic (ingress and egress) is denied by default. Pods without policy (or incorrect policy) are not allowed traffic until appropriate network policy is defined. Although the staging policy tool will help you find incorrect and missing policy, a global deny helps mitigate against other lateral malicious attacks. ## Best practice #1: Allow, stage, then deny[​](#best-practice-1-allow-stage-then-deny) We recommend that you create a global default deny policy *after you complete writing policy for the traffic that you want to allow*. Use the stage policy feature to get your allowed traffic working as expected, then lock down the cluster to block unwanted traffic. The following steps summarizes the best practice: 1. Create a staged global default deny policy. It will shows all the traffic that would be blocked if it were converted into a deny. 2. Create other network policies to individually allow the traffic shown as blocked in step 1, until no connections are denied. 3. Convert the staged global network policy to an enforced policy. ## Best practice #2: Keep the scope to non-system pods[​](#best-practice-2-keep-the-scope-to-non-system-pods) A global default deny policy applies to the entire cluster including all workloads in all namespaces, hosts (computers that run the hypervisor for VMs, or container runtime for containers), including Kubernetes control plane and Calico Cloud control plane nodes and pods. For this reason, the best practice is to create a global default deny policy for **non-system pods** as shown in the following example. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: deny-app-policy spec: namespaceSelector: has(projectcalico.org/name) && projectcalico.org/name not in {"kube-system", "calico-system"} types: - Ingress - Egress egress: # allow all namespaces to communicate to DNS pods - action: Allow protocol: UDP destination: selector: 'k8s-app == "kube-dns"' ports: - 53 - action: Allow protocol: TCP destination: selector: 'k8s-app == "kube-dns"' ports: - 53 ``` Note the following: - Even though we call this policy "global default deny", the above policy is not explicitly *denying traffic*. By selecting the traffic with the `namespaceSelector` but not specifying an allow, the traffic is denied after all other policy is evaluated. This design also makes it unnecessary to ensure any specific order (priority) for the default-deny policy. - Allowing access to `kube-dns` simplifies per-pod policies because you don't need to duplicate the DNS rules in every policy - The policy deliberately excludes the `kube-system` and `calico-system` namespaces by using a negative `namespaceSelector` to avoid impacting any control plane components Next, add the policy to the default tier. (As noted above, anywhere in the default tier is fine.) Next, use the stage policy feature and verify that the policy does not block any necessary traffic before enforcing it. ### Don't try this\![​](#dont-try-this) The following policy looks fine on the surface, and it does works. But as described in Best practices #2, the policy could break your cluster because the scope is too broad. Therefore, we do not recommend adding this type of policy to the default tier, even if you have verified allowed traffic using the stage policy feature. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: default.default-deny spec: tier: default selector: all() types: - Ingress - Egress ``` ### Platform application access controls In this article you will learn some best practices to secure platform tier applications. ## Isolate platform applications[​](#isolate-platform-applications) You may have several types of platform applications in your environment: - Storage platform - Secret management - Container security - Platform monitoring - Application performance monitoring We recommend that you implement all your platform applications controls in a **platform tier**, implicitly allowing granular ingress and egress controls for platform applications pods and denying everything else. This effectively isolates platform applications from business applications, and extends the multi-tenancy and application microsegmentation controls to platform applications. ## Platform order[​](#platform-order) ```yaml apiVersion: projectcalico.org/v3 kind: Tier metadata: name: platform spec: order: 200 ``` Order 200 implies that the platform tier comes before the security tier and that its policies are processed first. Some organizations require that security team controls take precedence over other controls. In that case, the security tier policies need to pass controls to the platform tier for granular control. ## Map your application access controls to policy[​](#map-your-application-access-controls-to-policy) Next, we recommend that you refer to your platform application documentation and get the communication map of application pods, and reflect the controls in your network policy. ## Create a global network policy[​](#create-a-global-network-policy) Here is our global network policy that allows application (`platform-app1`) and service (`platf-app1-svc2`), ingress access to service (`svc = platf-app1-svc1`) from port 12345. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: platform.platform-app2-svc2 spec: tier: platform selector: (app == "platform-app1" && svc == "platf-app1-svc2") ingress: - action: Allow protocol: TCP source: selector: svc == "platf-app1-svc1" destination: ports: - '12345' types: - Ingress ``` ### Kubernetes for beginners ## [📄️ What is network policy?](https://docs.tigera.io/calico-cloud/tutorials/training/about-network-policy) [Learn the basics of Kubernetes and Calico Cloud network policy](https://docs.tigera.io/calico-cloud/tutorials/training/about-network-policy) ## [📄️ Kubernetes services](https://docs.tigera.io/calico-cloud/tutorials/training/about-kubernetes-services) [Learn the three main service types and how to use them.](https://docs.tigera.io/calico-cloud/tutorials/training/about-kubernetes-services) ## [📄️ Kubernetes ingress](https://docs.tigera.io/calico-cloud/tutorials/training/about-kubernetes-ingress) [Learn the different ingress implementations and how ingress and policy interact.](https://docs.tigera.io/calico-cloud/tutorials/training/about-kubernetes-ingress) ## [📄️ Kubernetes egress](https://docs.tigera.io/calico-cloud/tutorials/training/about-kubernetes-egress) [Learn why you should restrict egress traffic and how to do it.](https://docs.tigera.io/calico-cloud/tutorials/training/about-kubernetes-egress) ### What is network policy? > **SECONDARY:** This guide provides educational material that is not specific to Calico Cloud. Kubernetes and Calico Cloud provide network policy APIs to help you secure your workloads. In this guide you will learn: - What network policy is and why it is important. - The differences between Kubernetes and Calico Cloud network policies and when you might want to use each. - Some best practices for using network policy. ## What is network policy?[​](#what-is-network-policy-1) Network policy is the primary tool for securing a Kubernetes network. It allows you to easily restrict the network traffic in your cluster so only the traffic that you want to flow is allowed. To understand the significance of network policy, let's briefly explore how network security was typically achieved prior to network policy. Historically in enterprise networks, network security was provided by designing a physical topology of network devices (switches, routers, firewalls) and their associated configuration. The physical topology defined the security boundaries of the network. In the first phase of virtualization, the same network and network device constructs were virtualized in the cloud, and the same techniques for creating specific network topologies of (virtual) network devices were used to provide network security. Adding new applications or services often required additional network design to update the network topology and network device configuration to provide the desired security. In contrast, the [Kubernetes network model](https://docs.tigera.io/calico-cloud/tutorials/training/about-kubernetes-networking) defines a "flat" network in which every pod can communicate with all other pods in the cluster using pod IP addresses. This approach massively simplifies network design and allows new workloads to be scheduled dynamically anywhere in the cluster with no dependencies on the network design. In this model, rather than network security being defined by network topology boundaries, it is defined using network policies that are independent of the network topology. Network policies are further abstracted from the network by using label selectors as their primary mechanism for defining which workloads can talk to which workloads, rather than IP addresses or IP address ranges. ## Why is network policy important?[​](#why-is-network-policy-important) In an age where attackers are becoming more and more sophisticated, network security as a line of defense is more important than ever. While you can (and should) use firewalls to restrict traffic at the perimeters of your network (commonly referred to as north-south traffic), their ability to police Kubernetes traffic is often limited to a granularity of the cluster as a whole, rather than to specific groups of pods, due to the dynamic nature of pod scheduling and pod IP addresses. In addition, the goal of most attackers once they gain a small foothold inside the perimeter is to move laterally (commonly referred to as east-west) to gain access to higher value targets, which perimeter based firewalls can't police against. Network policy on the other hand is designed for the dynamic nature of Kubernetes by following the standard Kubernetes paradigm of using label selectors to define groups of pods, rather than IP addresses. And because network policy is enforced within the cluster itself it can police both north-south and east-west traffic. Network policy represents an important evolution of network security, not just because it handles the dynamic nature of modern microservices, but because it empowers dev and devops engineers to easily define network security themselves, rather than needing to learn low-level networking details or raise tickets with a separate team responsible for managing firewalls. Network policy makes it easy to define intent, such as "only this microservice gets to connect to the database", write that intent as code (typically in YAML files), and integrate authoring of network policies into git workflows and CI/CD processes. > **SECONDARY:** Note: Calico Cloud offers capabilities that can help perimeter firewalls integrate more tightly with Kubernetes. However, this does not remove the need or value of network policies within the cluster itself.) ## Kubernetes network policy[​](#kubernetes-network-policy) Kubernetes network policies are defined using the Kubernetes [NetworkPolicy](https://kubernetes.io/docs/reference/kubernetes-api/policy-resources/network-policy-v1/) resource. The main features of Kubernetes network policies are: - Policies are namespace scoped (i.e. you create them within the context of a specific namespace just like, for example, pods) - Policies are applied to pods using label selectors - Policy rules can specify the traffic that is allowed to/from other pods, namespaces, or CIDRs - Policy rules can specify protocols (TCP, UDP, SCTP), named ports or port numbers Kubernetes itself does not enforce network policies, and instead delegates their enforcement to network plugins. Most network plugins implement the mainline elements of Kubernetes network policies, though not all implement every feature of the specification. (Calico Cloud does implement every feature, and was the original reference implementation of Kubernetes network policies.) ## Calico Cloud network policy[​](#calico-cloud-network-policy) In addition to enforcing Kubernetes network policy, Calico Cloud supports its own namespaced [NetworkPolicy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) and non-namespaced [GlobalNetworkPolicy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) resources, which provide additional features beyond those supported by Kubernetes network policy. This includes support for: - policy ordering/priority - deny and log actions in rules - more flexible match criteria for applying policies and in policy rules, including matching on Kubernetes ServiceAccounts, and (if using Istio & Envoy) cryptographic identity and layer 5-7 match criteria such as HTTP & gRPC URLs. - ability to reference non-Kubernetes workloads in polices, including matching on [NetworkSets](https://docs.tigera.io/calico-cloud/reference/resources/networkset) in policy rules While Kubernetes network policy applies only to pods, Calico Cloud network policy can be applied to multiple types of endpoints including pods, VMs, and host interfaces. To learn more about Calico Cloud network policies, read the [Get started with Calico Cloud network policy](https://docs.tigera.io/calico-cloud/network-policy/beginners/calico-network-policy) guide. ## Benefits of using Calico Cloud for network policy[​](#benefits-of-using-calico-cloud-for-network-policy) ### Full Kubernetes network policy support[​](#full-kubernetes-network-policy-support) Unlike some other network policy implementations, Calico Cloud implements the full set of Kubernetes network policy features. ### Richer network policy[​](#richer-network-policy) Calico Cloud network policies allow even richer traffic control than Kubernetes network policies if you need it. In addition, Calico Cloud network policies allow you to create policy that applies across multiple namespaces using GlobalNetworkPolicy resources. ### Mix Kubernetes and Calico Cloud network policy[​](#mix-kubernetes-and-calico-cloud-network-policy) Kubernetes and Calico Cloud network policies can be mixed together seamlessly. One common use case for this is to split responsibilities between security / cluster ops teams and developer / service teams. For example, giving the security / cluster ops team RBAC permissions to define Calico Cloud policies, and giving developer / service teams RBAC permissions to define Kubernetes network policies in their specific namespaces. As Calico Cloud policy rules can be ordered to be enforced either before or after Kubernetes network policies, and can include actions such as deny and log, this allows the security / cluster ops team to define basic higher-level more-general purpose rules, while empowering the developer / service teams to define their own fine-grained constraints on the apps and services they are responsible for. For more flexible control and delegation of responsibilities between two or more teams, Calico Cloud extends this model to provide hierarchical policy. ![Example mix of network policy types](https://docs.tigera.io/assets/images/example-k8s-calico-policy-mix-c22a42da6321a0be5f0963664fd5baf3.svg) ### Extendable with Calico Cloud[​](#extendable-with-calico-cloud) Calico Cloud adds even richer network policy capabilities, with the ability to specify hierarchical policies, with each team have particular boundaries of trust, and FQDN / domain names in policy rules for restricting access to specific external services. ## Best practices for network policies[​](#best-practices-for-network-policies) ### Ingress and egress[​](#ingress-and-egress) At a minimum we recommend that every pod is protected by network policy ingress rules that restrict what is allowed to connect to the pod and on which ports. The best practice is also to define network policy egress rules that restrict the outgoing connections that are allowed by pods themselves. Ingress rules protect your pod from attacks outside of the pod. Egress rules help protect everything outside of the pod if the pod gets compromised, reducing the attack surface to make moving laterally (east-west) or to prevent an attacker from exfiltrating compromised data from your cluster (north-south). ### Policy schemas[​](#policy-schemas) Due to the flexibility of network policy and labelling, there are often multiple different ways of labelling and writing policies that can achieve the same particular goal. One of the most common approaches is to have a small number of global policies that apply to all pods, and then a single pod specific policy that defines all the ingress and egress rules that are particular to that pod. For example: ```yaml kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: front-end namespace: staging spec: podSelector: matchLabels: app: back-end ingress: - from: - podSelector: matchLabels: app: front-end ports: - protocol: TCP port: 443 egress: - to: - podSelector: matchLabels: app: database ports: - protocol: TCP port: 27017 ``` ### Default deny[​](#default-deny) One approach to ensuring these best practices are being followed is to define [default deny](https://docs.tigera.io/calico-cloud/network-policy/beginners/kubernetes-default-deny) network policies. These ensure that if no other policy is defined that explicitly allows traffic to/from a pod, then the traffic will be denied. As a result, anytime a team deploys a new pod, they are forced to also define network policy for the pod. It can be useful to use a Calico Cloud GlobalNetworkPolicy for this (rather than needing to define a policy every time a new namespace is created) and to include some exceptions to the default deny (for example to allow pods to access DNS). For example, you might use the following policy to default-deny all (non-system) pod traffic except for DNS queries to kube-dns/core-dns. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: default-app-policy spec: namespaceSelector: has(projectcalico.org/name) && projectcalico.org/name not in {"kube-system", "calico-system"} types: - Ingress - Egress egress: - action: Allow protocol: UDP destination: selector: k8s-app == "kube-dns" ports: - 53 ``` ### Hierarchical policy[​](#hierarchical-policy) [Calico Cloud](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/tiered-policy) supports hierarchical network policy using policy tiers. RBAC for each tier can be defined to restrict who can interact with each tier. This can be used to delegate trust across multiple teams. ![Example tiers](https://docs.tigera.io/assets/images/example-tiers-350db070cf4f095db5de19ecbcc1e9ea.svg) ### Kubernetes services > **SECONDARY:** This guide provides educational material that is not specific to Calico Cloud. In this guide you will learn: - What are Kubernetes services? - What are the differences between the three main service types and what do you use them for? - How do services and network policy interact? - Some options for optimizing how services are handled. ## What are Kubernetes services?[​](#what-are-kubernetes-services) Kubernetes [Services](https://kubernetes.io/docs/concepts/services-networking/service/) provide a way of abstracting access to a group of pods as a network service. The group of pods backing each service is usually defined using a [label selector](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels). When a client connects to a Kubernetes service, the connection is load balanced to one of the pods backing the service, as illustrated in this conceptual diagram: ![Kubernetes Service conceptual diagram](https://docs.tigera.io/assets/images/k8s-service-concept-77a22aa6f22a4048f2aacaccca788656.svg) There are three main types of Kubernetes services: - Cluster IP - which is the usual way of accessing a service from inside the cluster - Node port - which is the most basic way of accessing a service from outside the cluster - Load balancer - which uses an external load balancer as a more sophisticated way to access a service from outside the cluster. ## Cluster IP services[​](#cluster-ip-services) The default service type is `ClusterIP`. This allows a service to be accessed within the cluster via a virtual IP address, known as the service Cluster IP. The Cluster IP for a service is discoverable through Kubernetes DNS. For example, `my-svc.my-namespace.svc.cluster-domain.example`. The DNS name and Cluster IP address remain constant for the life time of the service, even though the pods backing the service may be created or destroyed, and the number of pods backing the service may change over time. In a typical Kubernetes deployment, kube-proxy runs on every node and is responsible for intercepting connections to Cluster IP addresses and load balancing across the group of pods backing each service. As part of this process [DNAT](https://docs.tigera.io/calico-cloud/tutorials/training/about-networking) is used to map the destination IP address from the Cluster IP to the chosen backing pod. Response packets on the connection then have the NAT reverse on their way back to the pod that initiated the connection. ![kube-proxy cluster IP](https://docs.tigera.io/assets/images/kube-proxy-cluster-ip-da828a9e8ef7c825261c2a87852ed1ac.svg) Importantly, network policy is enforced based on the pods, not the service Cluster IP. (i.e. Egress network policy is enforced for the client pod after the DNAT has changed the connection's destination IP to the chosen service backing pod. And because only the destination IP for the connection is changed, ingress network policy for the backing pod sees the original client pod as the source of the connection.) ## Node port services[​](#node-port-services) The most basic way to access a service from outside the cluster is to use a service of type `NodePort`. A Node Port is a port reserved on each node in the cluster through which the service can be accessed. In a typical Kubernetes deployment, kube-proxy is responsible for intercepting connections to Node Ports and load balancing them across the pods backing each service. As part of this process NAT is used to map the destination IP address and port from the node IP and Node Port, to the chosen backing pod and service port. In addition the source IP address is mapped from the client IP to the node IP, so that response packets on the connection flow back via the original node, where the NAT can be reversed. (It's the node which performed the NAT that has the connection tracking state needed to reverse the NAT.) ![kube-proxy node port](https://docs.tigera.io/assets/images/kube-proxy-node-port-b8daa0296e4e25f65be4892bc45c7fae.svg) Note that because the connection source IP address is SNATed to the node IP address, ingress network policy for the service backing pod does not see the original client IP address. Typically this means that any such policy is limited to restricting the destination protocol and port, and cannot restrict based on the client / source IP. This limitation can be circumvented in some scenarios by using [externalTrafficPolicy](#externaltrafficpolicylocal) or by using Calico Cloud's eBPF data plane native service handling (rather than kube-proxy) which preserves source IP address. ## Load balancer services[​](#load-balancer-services) Services of type `LoadBalancer` expose the service via an external network load balancer (NLB). The exact type of network load balancer depends on which public cloud provider or, if on-prem, which specific hardware load balancer integration is integrated with your cluster. The service can be accessed from outside of the cluster via a specific IP address on the network load balancer, which by default will load balance evenly across the nodes using the service node port. ![kube-proxy load balancer](https://docs.tigera.io/assets/images/kube-proxy-load-balancer-07d81e8c44b28b285157e15f8a75f2b4.svg) Most network load balancers preserve the client source IP address, but because the service then goes via a node port, the backing pods themselves do not see the client IP, with the same implications for network policy. As with node ports, this limitation can be circumvented in some scenarios by using [externalTrafficPolicy](#externaltrafficpolicylocal) or by using Calico Cloud's eBPF data plane [native service handling](#calico-ebpf-native-service-handling) (rather than kube-proxy) which preserves source IP address. ## Advertising service IPs[​](#advertising-service-ips) One alternative to using node ports or network load balancers is to advertise service IP addresses over BGP. This requires the cluster to be running on an underlying network that supports BGP, which typically means an on-prem deployment with standard Top of Rack routers. Calico Cloud supports advertising service Cluster IPs, or External IPs for services configured with one. If you are not using Calico as your network plugin then [MetalLB](https://github.com/metallb/metallb) provides similar capabilities that work with a variety of different network plugins. ![kube-proxy service advertisement](https://docs.tigera.io/assets/images/kube-proxy-service-advertisement-602a7370d80c58d8f6b64480b88626ea.svg) ## externalTrafficPolicy:local[​](#externaltrafficpolicy) By default, whether using service type `NodePort` or `LoadBalancer` or advertising service IP addresses over BGP, accessing a service from outside the cluster load balances evenly across all the pods backing the service, independent of which node the pods are on. This behavior can be changed by configuring the service with `externalTrafficPolicy:local` which specifies that connections should only be load balanced to pods backing the service on the local node. When combined with services of type `LoadBalancer` or with Calico Cloud service IP address advertising, traffic is only directed to nodes that host at least one pod backing the service. This reduces the potential extra network hop between nodes, and perhaps more importantly, to maintain the source IP address all the way to the pod, so network policy can restrict specific external clients if desired. ![kube-proxy service advertisement](https://docs.tigera.io/assets/images/kube-proxy-service-local-fd625e0b06b7ae8b5a65ffc4326768b3.svg) Note that in the case of services of type `LoadBalancer`, not all Load Balancers support this mode. And in the case of service IP advertisement, the evenness of the load balancing becomes topology dependent. In this case, pod anti-affinity rules can be used to ensure even distribution of backing pods across your topology, but this does add some complexity to deploying the service. ## Calico eBPF native service handling[​](#calico-ebpf-native-service-handling) As an alternative to using Kubernetes standard kube-proxy, Calico Cloud's eBPF data plane supports native service handling. This preserves source IP to simplify network policy, offers DSR (Direct Server Return) to reduce the number of network hops for return traffic, and provides even load balancing independent of topology, with reduced CPU and latency compared to kube-proxy. ![kube-proxy service advertisement](https://docs.tigera.io/assets/images/calico-native-service-handling-2c372633cfc26035dd0a13b3dfd61ac5.svg) # Additional resources - [Video: Everything you need to know about Kubernetes Services networking](https://www.projectcalico.org/everything-you-need-to-know-about-kubernetes-services-networking/) - [Blog: Introducing the Calico eBPF data plane](https://www.projectcalico.org/introducing-the-calico-ebpf-dataplane/) - [Blog: Hands on with Calico eBPF native service handling](https://www.projectcalico.org/hands-on-with-calicos-ebpf-service-handling/) ### Kubernetes ingress > **SECONDARY:** This guide provides education material that is not specific to Calico Cloud. In this guide you will learn: - What is Kubernetes ingress? - Why use ingress? - What are the differences between different ingress implementations? - How does ingress and network policy interact? - How does ingress and services fit together under the covers? ## What is Kubernetes ingress?[​](#what-is-kubernetes-ingress) Kubernetes Ingress builds on top of Kubernetes [Services](https://docs.tigera.io/calico-cloud/tutorials/training/about-kubernetes-services) to provide load balancing at the application layer, mapping HTTP and HTTPS requests with particular domains or URLs to Kubernetes services. Ingress can also be used to terminate SSL / TLS before load balancing to the service. The details of how Ingress is implemented depend on which [Ingress Controller](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/) you are using. The Ingress Controller is responsible for monitoring Kubernetes [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) resources and provisioning / configuring one or more ingress load balancers to implement the desired load balancing behavior. Unlike Kubernetes services, which are handled at the network layer (L3-4), ingress load balancers operate at the application layer (L5-7). Incoming connections are terminated at the load balancer so it can inspect the individual HTTP / HTTPS requests. The requests are then forwarded via separate connections from the load balancer to the chosen service backing pods. As a result, network policy applied to the backing pods can restrict access to only allow connections from the load balancer, but cannot restrict access to specific original clients. ## Why use Kubernetes Ingress?[​](#why-use-kubernetes-ingress) Given that Kubernetes [Services](https://docs.tigera.io/calico-cloud/tutorials/training/about-kubernetes-services) already provide a mechanism for load balancing access to services from outside of the cluster, why might you want to use Kubernetes Ingress? The mainline use case is if you have multiple HTTP / HTTPS services that you want to expose through a single external IP address, perhaps with each service having a different URL path, or perhaps as multiple different domains. This is lot simpler from a client configuration point of view than exposing each service outside of the cluster using Kubernetes Services, which would give each service a separate external IP address. If on the other hand, your application architecture is fronted by a single "front end" microservice then Kubernetes Services likely already meet your needs. In this case you might prefer to not add Ingress to the picture, both from a simplicity point of view, and potentially also so you can more easily restrict access to specific clients using network policy. In effect, your "front end" microservice already plays the role of Kubernetes Ingress, in a way that is not that dissimilar to [in-cluster ingress](#in-cluster-ingress-solutions) solutions discussed below. ## Types of Ingress solutions[​](#types-of-ingress-solutions) Broadly speaking there are two types of ingress solutions: - In-cluster ingress - where ingress load balancing is performed by pods within the cluster itself. - External ingress - where ingress load balancing is implemented outside of the cluster by appliances or cloud provider capabilities. ### In-cluster ingress solutions[​](#in-cluster-ingress-solutions) In-cluster ingress solutions use software load balancers running in pods within the cluster itself. There are many different ingress controllers to consider that follow this pattern, including for example the NGINX ingress controller. The advantages of this approach are that you can: - horizontally scale your ingress solution up to the limits of Kubernetes - choose the ingress controller that best suits your specific needs, for example, with particular load balancing algorithms, or security options. To get your ingress traffic to the in-cluster ingress pods, the ingress pods are normally exposed externally as a Kubernetes service, so you can use any of the standard ways of accessing the service from outside of the cluster. A common approach is use an external network load balancer or service IP advertisement, with `externalTrafficPolicy:local`. This minimizes the number of network hops, and retains the client source IP address, which allows network policy to be used to restrict access to the ingress pods to particular clients if desired. ![In-cluster ingress](https://docs.tigera.io/assets/images/ingress-in-cluster-cd15c60b9423ec081e33c35221e091b7.svg) ### External ingress solutions[​](#external-ingress-solutions) External ingress solutions use application load balancers outside of the cluster. The exact details and features depend on which ingress controller you are using, but most cloud providers include an ingress controller that automates the provisioning and management of the cloud provider's application load balancers to provide ingress. The advantages of this type of ingress solution is that your cloud provider handles the operational complexity of the ingress for you. The downsides are a potentially more limited set of features compared to the rich range of in-cluster ingress solutions, and the maximum number of services exposed by ingress being constrained by cloud provider specific limits. ![External ingress](https://docs.tigera.io/assets/images/ingres-external-61762bf0d132a8eb2ab1080e1b65bac3.svg) Note that most application load balancers support a basic mode of operation of forwarding traffic to the chosen service backing pods via the [node port](https://docs.tigera.io/calico-cloud/tutorials/training/about-kubernetes-services#node-port-services) of the corresponding service. In addition to this basic approach of load balancing to service node ports, some cloud providers support a second mode of application layer load balancing, which load balances directly to the pods backing each service, without going via node-ports or other kube-proxy service handling. This has the advantage of eliminating the potential second network hop associated with node ports load balancing to a pod on a different node. The potential disadvantage is that if you are operating at very high scales, for example with hundreds of pods backing a service, you may exceed the application layer load balancers maximum limit of IPs it can load balance to in this mode. In this case switching to an in-cluster ingress solution is likely the better fit for you. ## Show me everything\![​](#show-me-everything) All the above diagrams focus on connection level (L5-7) representation of ingress and services. You can learn more about the network level (L3-4) interactions involved in handling the connections, including which scenarios client source IP addresses are maintained, in the [About Kubernetes Services](https://docs.tigera.io/calico-cloud/tutorials/training/about-kubernetes-services) guide. If you are already up to speed on how services work under the covers, here are some more complete diagrams that show details of how services are load balanced at the network layer (L3-4). > **SECONDARY:** You can successfully use ingress without needing to understand this next level of detail! So feel free to skip over these diagrams if you don't want to dig deeper into how services and ingress interact under the covers. **In-cluster ingress solution exposed as service type `LoadBalancer` with `externalTrafficPolicy:local`** ![In-cluster ingress with NLB local](https://docs.tigera.io/assets/images/ingress-in-cluster-nlb-local-7524c660d10f21f3f7875eb553ca37f0.svg) **External ingress solution via node ports** ![External ingress via node port](https://docs.tigera.io/assets/images/ingress-external-node-ports-c8e0de7aec96801c340be6436d52d6ab.svg) **External ingress solution direct to pods** ![External ingress direct to pods](https://docs.tigera.io/assets/images/ingress-external-direct-to-pods-95ae5150ca478ff2b87b750586f19c6b.svg) ## Additional resources[​](#additional-resources) - [Video: Everything you need to know about Kubernetes Ingress networking](https://www.projectcalico.org/everything-you-need-to-know-about-kubernetes-ingress-networking/) - [Video: Everything you need to know about Kubernetes Services networking](https://www.projectcalico.org/everything-you-need-to-know-about-kubernetes-services-networking/) ### Kubernetes egress > **SECONDARY:** This guide provides educational material that is not specific to Calico Cloud. In this guide you will learn: - What is Kubernetes egress? - Why should you restrict egress traffic and how can you do it? - What is "NAT outgoing" and when is it used? - What is an egress gateway, and why might you want to use one? ## What is Kubernetes egress?[​](#what-is-kubernetes-egress) In this guide we are using the term Kubernetes egress to describe connections being made from pods to anything outside of the cluster. In contrast to ingress traffic, where Kubernetes has the [Ingress](https://docs.tigera.io/calico-cloud/tutorials/training/about-kubernetes-ingress) resource type to help manage the traffic, there is no Kubernetes Egress resource. Instead, how the egress traffic is handled at a networking level is determined by the Kubernetes network implementation / CNI plugin being used by the cluster. In addition, if a service mesh is being used, this can add egress behaviors on top of those the network implementation provides. There are three areas of behavior worth understanding for egress traffic, so you can choose a networking and/or service mesh setup that best suits your needs: - Restricting egress traffic - Outgoing NAT behavior - Egress gateways ## Restricting egress traffic[​](#restricting-egress-traffic) It's a common security requirement and best practice to restrict outgoing connections from the cluster. This is normally achieved using [Network Policy](https://docs.tigera.io/calico-cloud/tutorials/training/about-network-policy) to define egress rules for each microservice, often in conjunction with a [default deny](https://docs.tigera.io/calico-cloud/tutorials/training/about-network-policy#default-deny) policy that ensures outgoing connections are denied by default, until a policy is defined to explicitly allow specific traffic. One limitation when using Kubernetes Network Policy to restrict access to specific external resources, is that the external resources need to be specified as IP addresses (or IP address ranges) within the policy rules. If the IP addresses associated with an external resource change, then every policy that referenced those IP addresses needs to be updated with the new IP addresses. This limitation can be circumvented using Calico Cloud [Use external IPs or networks rules in policy](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/external-ips-policy), or [DNS policy](https://docs.tigera.io/calico-cloud/network-policy/domain-based-policy) in policy rules. In addition to using network policy, service meshes typically allow you to configure which external services each pod can access. In the case of Istio, Calico Cloud can be integrated to enforce network policy at the service mesh layer, including L5-7 rules, as another alternative to using IP addresses in rules. Note in addition to everything mentioned so far, perimeter firewalls can also be used to restrict outgoing connections, for example to allow connections only to particular external IP address ranges, or external services. However, since perimeter firewalls typically cannot distinguish individual pods, the rules apply equally to all pods in the cluster. This provides some defense in depth, but cannot replace the requirement for network policy. ## NAT outgoing[​](#nat-outgoing) Network Address Translation ([NAT](https://en.wikipedia.org/wiki/Network_address_translation)) is the process of mapping an IP address in a packet to a different IP address as the packet passes through the device performing the NAT. Depending on the use case, NAT can apply to the source or destination IP address, or to both addresses. In the context of Kubernetes egress, NAT is used to allow pods to connect to services outside of the cluster if the pods have IP addresses that are not routable outside of the cluster (for example, if the pod network is an overlay). For example, if a pod in an overlay network attempts to connect to an IP address outside of the cluster, then the node hosting the pod uses SNAT (Source Network Address Translation) to map the non-routable source IP address of the packet to the node's IP address before forwarding on the packet. The node then maps response packets coming in the opposite direction back to the original pod IP address, so packets flow end-to-end in both directions, with neither pod or external service being aware the mapping is happening. In most clusters this NAT behavior is configured statically across the whole of the cluster. When using Calico Cloud, the NAT behavior can be configured at a more granular level for particular address ranges using [IP pools](https://docs.tigera.io/calico-cloud/reference/resources/ippool) . This effectively allows the scope of "non-routable" to be more tightly defined than just "inside the cluster vs outside the cluster", which can be useful in some enterprise deployment scenarios. ## Egress gateways[​](#egress-gateways) Another approach to Kubernetes egress is to route all outbound connections via one or more egress gateways. The gateways SNAT (Source Network Address Translation) the connections so the external service being connected to sees the connection as coming from the egress gateway. The main use case is to improve security, either with the egress gateway performing a direct security role in terms of what connections it allows, or in conjunction with perimeter firewalls (or other external entities). For example, so that perimeter firewalls see the connections coming from well known IP addresses (the egress gateways) rather than from dynamic pod IP addresses they don't understand. Egress gateways are not a native concept in Kubernetes itself, but are implemented by some Kubernetes network implementations and some service meshes. For example, Calico Cloud provides egress gateway functionality, plus the ability to map namespaces (or even individual pods) to specific egress gateways. Perimeter firewalls (or other external security entities) can then effectively provide per namespace security controls, even though they do not have visibility to dynamic pod IP addresses. As an alternative approach to egress gateways, Calico Cloud allows you to control pod IP address ranges based on namespace, or node, or even at the individual pod level. Assuming no outgoing NAT is required, this provides a very simple way for perimeter firewalls (or other external security entities) to integrate with Kubernetes for both ingress and egress traffic. (Note that this approach relies on having enough address space available to sensibly assign IP address ranges, for example to each namespace, so it can lead to IP address range exhaustion challenges for large scale deployments. In these scenarios, using egress gateways is likely to be a better option.) ## Additional resources[​](#additional-resources) - [Use external IPs or networks rules in policy](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/external-ips-policy) - [Restrict a pod to use an IP address in a specific range](https://docs.tigera.io/calico-cloud/networking/ipam/legacy-firewalls) - [Assign IP addresses based on topology](https://docs.tigera.io/calico-cloud/networking/ipam/assign-ip-addresses-topology) - [Egress gateways](https://docs.tigera.io/calico-cloud/networking/egress/) ### Kubernetes tutorials and demos ## [📄️ Kubernetes policy, demo](https://docs.tigera.io/calico-cloud/tutorials/kubernetes-tutorials/kubernetes-demo) [An interactive demo that visually shows how applying Kubernetes policy allows and denies connections.](https://docs.tigera.io/calico-cloud/tutorials/kubernetes-tutorials/kubernetes-demo) ## [📄️ Get started with Kubernetes network policy](https://docs.tigera.io/calico-cloud/tutorials/kubernetes-tutorials/kubernetes-network-policy) [Learn Kubernetes policy syntax, rules, and features for controlling network traffic.](https://docs.tigera.io/calico-cloud/tutorials/kubernetes-tutorials/kubernetes-network-policy) ## [📄️ Kubernetes policy, basic tutorial](https://docs.tigera.io/calico-cloud/tutorials/kubernetes-tutorials/kubernetes-policy-basic) [Learn how to use basic Kubernetes network policy to securely restrict traffic to/from pods.](https://docs.tigera.io/calico-cloud/tutorials/kubernetes-tutorials/kubernetes-policy-basic) ## [📄️ Kubernetes policy, advanced tutorial](https://docs.tigera.io/calico-cloud/tutorials/kubernetes-tutorials/kubernetes-policy-advanced) [Learn how to create more advanced Kubernetes network policies (namespace, allow and deny all ingress and egress).](https://docs.tigera.io/calico-cloud/tutorials/kubernetes-tutorials/kubernetes-policy-advanced) ### Kubernetes policy, demo The included demo sets up a frontend and backend service, as well as a client service, all running on Kubernetes. It then configures network policy on each service. ## Running the stars example[​](#running-the-stars-example) ### 1) Create the frontend, backend, client, and management-ui apps.[​](#1-create-the-frontend-backend-client-and-management-ui-apps) ```shell kubectl create -f https://docs.tigera.io/files/00-namespace.yaml kubectl create -f https://docs.tigera.io/files/01-management-ui.yaml kubectl create -f https://docs.tigera.io/files/02-backend.yaml kubectl create -f https://docs.tigera.io/files/03-frontend.yaml kubectl create -f https://docs.tigera.io/files/04-client.yaml ``` Wait for all the pods to enter `Running` state. ```bash kubectl get pods --all-namespaces --watch ``` > Note that it may take several minutes to download the necessary Docker images for this demo. The management UI runs as a `NodePort` Service on Kubernetes, and shows the connectivity of the Services in this example. You can view the UI by visiting `http://:30002` in a browser. Once all the pods are started, they should have full connectivity. You can see this by visiting the UI. Each service is represented by a single node in the graph. - `backend` -> Node "B" - `frontend` -> Node "F" - `client` -> Node "C" ### 2) Enable isolation[​](#2-enable-isolation) Running following commands will prevent all access to the frontend, backend, and client Services. ```shell kubectl create -n stars -f https://docs.tigera.io/files/default-deny.yaml kubectl create -n client -f https://docs.tigera.io/files/default-deny.yaml ``` #### Confirm isolation[​](#confirm-isolation) Refresh the management UI (it may take up to 10 seconds for changes to be reflected in the UI). Now that we've enabled isolation, the UI can no longer access the pods, and so they will no longer show up in the UI. ### 3) Allow the UI to access the services using network policy objects[​](#3-allow-the-ui-to-access-the-services-using-network-policy-objects) Apply the following YAML files to allow access from the management UI. ```shell kubectl create -f https://docs.tigera.io/files/allow-ui.yaml kubectl create -f https://docs.tigera.io/files/allow-ui-client.yaml ``` After a few seconds, refresh the UI - it should now show the Services, but they should not be able to access each other anymore. ### 4) Create the backend-policy.yaml file to allow traffic from the frontend to the backend[​](#4-create-the-backend-policyyaml-file-to-allow-traffic-from-the-frontend-to-the-backend) ```shell kubectl create -f https://docs.tigera.io/files/backend-policy.yaml ``` Refresh the UI. You should see the following: - The frontend can now access the backend (on TCP port 6379 only). - The backend cannot access the frontend at all. - The client cannot access the frontend, nor can it access the backend. ### 5) Expose the frontend service to the client namespace[​](#5-expose-the-frontend-service-to-the-client-namespace) ```shell kubectl create -f https://docs.tigera.io/files/frontend-policy.yaml ``` The client can now access the frontend, but not the backend. Neither the frontend nor the backend can initiate connections to the client. The frontend can still access the backend. To use Calico Cloud to enforce egress policy on Kubernetes pods, see [the advanced policy demo](https://docs.tigera.io/calico-cloud/tutorials/kubernetes-tutorials/kubernetes-policy-advanced). ### 6) (Optional) Clean up the demo environment[​](#6-optional-clean-up-the-demo-environment) You can clean up the demo by deleting the demo Namespaces: ```bash kubectl delete ns client stars management-ui ``` ### Get started with Kubernetes network policy ## Big picture[​](#big-picture) Kubernetes network policy lets administrators and developers enforce which network traffic is allowed using rules. ## Value[​](#value) Kubernetes network policy lets developers secure access to and from their applications using the same simple language they use to deploy them. Developers can focus on their applications without understanding low-level networking concepts. Enabling developers to easily secure their applications using network policies supports a shift left DevOps environment. ## Concepts[​](#concepts) The Kubernetes Network Policy API provides a standard way for users to define network policy for controlling network traffic. However, Kubernetes has no built-in capability to enforce the network policy. To enforce network policy, you must use a network plugin such as Calico. ### Ingress and egress[​](#ingress-and-egress) The bulk of securing network traffic typically revolves around defining egress and ingress rules. From the point of view of a Kubernetes pod, **ingress** is incoming traffic to the pod, and **egress** is outgoing traffic from the pod. In Kubernetes network policy, you create ingress and egress “allow” rules independently (egress, ingress, or both). ### Default deny/allow behavior[​](#default-denyallow-behavior) **Default allow** means all traffic is allowed by default, unless otherwise specified. **Default deny** means all traffic is denied by default, unless explicitly allowed. ## How to[​](#how-to) Before you create your first Kubernetes network policy, you need to understand the default network policy behaviors. If no Kubernetes network policies apply to a pod, then all traffic to/from the pod are allowed (default-allow). As a result, if you do not create any network policies, then all pods are allowed to communicate freely with all other pods. If one or more Kubernetes network policies apply to a pod, then only the traffic specifically defined in that network policy are allowed (default-deny). You are now ready to start fine-tuning traffic that should be allowed. - [Create ingress policies](#create-ingress-policies) - [Allow ingress traffic from pods in the same namespace](#allow-ingress-traffic-from-pods-in-the-same-namespace) - [Allow ingress traffic from pods in a different namespace](#allow-ingress-traffic-from-pods-in-a-different-namespace) - [Create egress policies](#create-egress-policies) - [Allow egress traffic from pods in the same namespace](#allow-egress-traffic-from-pods-in-the-same-namespace) - [Allow egress traffic to IP addresses or CIDR range](#allow-egress-traffic-to-ip-addresses-or-cidr-range) - [Best practice: create deny-all default network policy](#best-practice-create-deny-all-default-network-policy) - [Create deny-all default ingress and egress network policy](#create-deny-all-default-ingress-and-egress-network-policy) ### Create ingress policies[​](#create-ingress-policies) Create ingress network policies to allow inbound traffic from other pods. Network policies apply to pods within a specific **namespace**. Policies can include one or more ingress rules. To specify which pods in the namespace the network policy applies to, use a **pod selector**. Within the ingress rule, use another pod selector to define which pods allow incoming traffic, and the **ports** field to define on which ports traffic is allowed. #### Allow ingress traffic from pods in the same namespace[​](#allow-ingress-traffic-from-pods-in-the-same-namespace) In the following example, incoming traffic to pods with label **color=blue** are allowed only if they come from a pod with **color=red**, on port **80**. ```yaml kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: allow-same-namespace namespace: default spec: podSelector: matchLabels: color: blue ingress: - from: - podSelector: matchLabels: color: red ports: - port: 80 ``` #### Allow ingress traffic from pods in a different namespace[​](#allow-ingress-traffic-from-pods-in-a-different-namespace) To allow traffic from pods in a different namespace, use a namespace selector in the ingress policy rule. In the following policy, the namespace selector matches one or more Kubernetes namespaces and is combined with the pod selector that selects pods within those namespaces. > **SECONDARY:** Namespace selectors can be used only in policy rules. The **spec.podSelector** applies to pods only in the same namespace as the policy. In the following example, incoming traffic is allowed only if they come from a pod with label **color=red**, in a namespace with label **shape=square**, on port **80**. ```yaml kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: allow-different-namespace namespace: default spec: podSelector: matchLabels: color: blue ingress: - from: - podSelector: matchLabels: color: red namespaceSelector: matchLabels: shape: square ports: - port: 80 ``` ### Create egress policies[​](#create-egress-policies) Create egress network policies to allow outbound traffic from pods. #### Allow egress traffic from pods in the same namespace[​](#allow-egress-traffic-from-pods-in-the-same-namespace) The following policy allows pod outbound traffic to other pods in the same namespace that match the pod selector. In the following example, outbound traffic is allowed only if they go to a pod with label **color=red**, on port **80**. ```yaml kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: allow-egress-same-namespace namespace: default spec: podSelector: matchLabels: color: blue egress: - to: - podSelector: matchLabels: color: red ports: - port: 80 ``` #### Allow egress traffic to IP addresses or CIDR range[​](#allow-egress-traffic-to-ip-addresses-or-cidr-range) Egress policies can also be used to allow traffic to specific IP addresses and CIDR ranges. Typically, IP addresses/ranges are used to handle traffic that is external to the cluster for static resources or subnets. The following policy allows egress traffic to pods in CIDR, **172.18.0.0/24**. ```yaml kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: allow-egress-external namespace: default spec: podSelector: matchLabels: color: red egress: - to: - ipBlock: cidr: 172.18.0.0/24 ``` ### Best practice: create deny-all default network policy[​](#best-practice-create-deny-all-default-network-policy) To ensure that all pods in the namespace are secure, a best practice is to create a default network policy. This avoids accidentally exposing an app or version that doesn’t have policy defined. #### Create deny-all default ingress and egress network policy[​](#create-deny-all-default-ingress-and-egress-network-policy) The following network policy implements a default **deny-all** ingress and egress policy, which prevents all traffic to/from pods in the **policy-demo** namespace. Note that the policy applies to all pods in the policy-demo namespace, but does not explicitly allow any traffic. All pods are selected, but because the default changes when pods are selected by a network policy, the result is: **deny all ingress and egress traffic**. (Unless the traffic is allowed by another network policy). ```yaml kind: NetworkPolicy apiVersion: networking.k8s.io/v1 metadata: name: default-deny namespace: policy-demo spec: podSelector: matchLabels: {} policyTypes: - Ingress - Egress ``` ## Additional resources[​](#additional-resources) - [Kubernetes Network Policy API documentation](https://v1-21.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#networkpolicyspec-v1-networking-k8s-io) ### Kubernetes policy, basic tutorial This guide provides a simple way to try out Kubernetes `NetworkPolicy` with Calico Cloud. It requires a Kubernetes cluster configured with Calico Cloud networking, and expects that you have `kubectl` configured to interact with the cluster. ## Configure namespaces[​](#configure-namespaces) This guide will deploy pods in a Kubernetes namespace. Let's create the `Namespace` object for this guide. ```bash kubectl create ns policy-demo ``` ## Create demo pods[​](#create-demo-pods) We'll use Kubernetes `Deployment` objects to easily create pods in the namespace. 1. Create some nginx pods in the `policy-demo` namespace. ```bash kubectl create deployment --namespace=policy-demo nginx --image=nginx ``` 2. Expose them through a service. ```bash kubectl expose --namespace=policy-demo deployment nginx --port=80 ``` 3. Ensure the nginx service is accessible. ```bash kubectl run --namespace=policy-demo access --rm -ti --image busybox /bin/sh ``` This should open up a shell session inside the `access` pod, as shown below. ```text Waiting for pod policy-demo/access-472357175-y0m47 to be running, status is Pending, pod ready: false If you don't see a command prompt, try pressing enter. / # ``` 4. From inside the `access` pod, attempt to reach the `nginx` service. ```bash wget -q nginx -O - ``` You should see a response from `nginx`. Great! Our service is accessible. You can exit the pod now. ## Enable isolation[​](#enable-isolation) Let's turn on isolation in our `policy-demo` namespace. Calico Cloud will then prevent connections to pods in this namespace. Running the following command creates a NetworkPolicy which implements a default deny behavior for all pods in the `policy-demo` namespace. ```bash kubectl create -f - < **SECONDARY:** The NetworkPolicy allows traffic from Pods with the label `run: access` to Pods with the label `app: nginx`. The labels are automatically added by kubectl and are based on the name of the resource. We should now be able to access the service from the `access` pod. ```bash kubectl run --namespace=policy-demo access --rm -ti --image busybox /bin/sh ``` This should open up a shell session inside the `access` pod, as shown below. ```text Waiting for pod policy-demo/access-472357175-y0m47 to be running, status is Pending, pod ready: false If you don't see a command prompt, try pressing enter. / # ``` Now from within the busybox `access` pod execute the following command to test access to the nginx service. ```bash wget -q --timeout=5 nginx -O - ``` However, we still cannot access the service from a pod without the label `run: access`. We can verify this as follows. ```bash kubectl run --namespace=policy-demo cant-access --rm -ti --image busybox /bin/sh ``` This should open up a shell session inside the `cant-access` pod, as shown below. ```text Waiting for pod policy-demo/cant-access-472357175-y0m47 to be running, status is Pending, pod ready: false If you don't see a command prompt, try pressing enter. / # ``` Now from within the busybox `cant-access` pod execute the following command to test access to the nginx service. ```bash wget -q --timeout=5 nginx -O - ``` The request should time out. ```text wget: download timed out / # ``` You can clean up the demo by deleting the demo namespace. ```bash kubectl delete ns policy-demo ``` This was just a simple example of the Kubernetes NetworkPolicy API and how Calico can secure your Kubernetes cluster. For more information on network policy in Kubernetes, see the [Kubernetes user-guide](https://kubernetes.io/docs/concepts/services-networking/network-policies/). For a slightly more detailed demonstration of policy, check out the [Kubernetes policy demo](https://docs.tigera.io/calico-cloud/tutorials/kubernetes-tutorials/kubernetes-demo). ### Kubernetes policy, advanced tutorial The Kubernetes `NetworkPolicy` API allows users to express ingress and egress policies (starting with Kubernetes 1.8.0) to Kubernetes pods based on labels and ports. This guide walks through using Kubernetes `NetworkPolicy` to define more complex network policies. ## Requirements[​](#requirements) - A working Kubernetes cluster and access to it using kubectl - Your Kubernetes nodes have connectivity to the public internet - You are familiar with [Kubernetes NetworkPolicy](https://docs.tigera.io/calico-cloud/tutorials/kubernetes-tutorials/kubernetes-policy-basic) ## Tutorial flow[​](#tutorial-flow) 1. Create the Namespace and Nginx Service 2. Deny all ingress traffic 3. Allow ingress traffic to Nginx 4. Deny all egress traffic 5. Allow egress traffic to kube-dns 6. Cleanup Namespace ## 1. Create the namespace and nginx service[​](#1-create-the-namespace-and-nginx-service) We'll use a new namespace for this guide. Run the following commands to create it and a plain nginx service listening on port 80. ```bash kubectl create ns advanced-policy-demo kubectl create deployment --namespace=advanced-policy-demo nginx --image=nginx kubectl expose --namespace=advanced-policy-demo deployment nginx --port=80 ``` ### Verify access - allowed all ingress and egress[​](#verify-access---allowed-all-ingress-and-egress) Open up a second shell session which has `kubectl` connectivity to the Kubernetes cluster and create a busybox pod to test policy access. This pod will be used throughout this tutorial to test policy access. ```bash kubectl run --namespace=advanced-policy-demo access --rm -ti --image busybox /bin/sh ``` This should open up a shell session inside the `access` pod, as shown below. ```text Waiting for pod advanced-policy-demo/access-472357175-y0m47 to be running, status is Pending, pod ready: false If you don't see a command prompt, try pressing enter. / # ``` Now from within the busybox "access" pod execute the following command to test access to the nginx service. ```bash wget -q --timeout=5 nginx -O - ``` It should return the HTML of the nginx welcome page. Still within the busybox "access" pod, issue the following command to test access to google.com. ```bash wget -q --timeout=5 google.com -O - ``` It should return the HTML of the google.com home page. ## 2. Deny all ingress traffic[​](#2-deny-all-ingress-traffic) Enable ingress isolation on the namespace by deploying a [default deny all ingress traffic policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/#default-deny-all-ingress-traffic). ```bash kubectl create -f - < Welcome to nginx!... ``` After creating the policy, we can now access the nginx Service. ## 4. Deny all egress traffic[​](#4-deny-all-egress-traffic) Enable egress isolation on the namespace by deploying a [default deny all egress traffic policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/#default-deny-all-egress-traffic). ```bash kubectl create -f - < **SECONDARY:** The `nslookup` command can take a minute or more to timeout. ## 5. Allow DNS egress traffic[​](#5-allow-dns-egress-traffic) Run the following to create a label of `name: kube-system` on the `kube-system` namespace and a `NetworkPolicy` which allows DNS egress traffic from any pods in the `advanced-policy-demo` namespace to the `kube-system` namespace. ```bash kubectl label namespace kube-system name=kube-system kubectl create -f - < Welcome to nginx!... ``` Next, try to retrieve the home page of google.com. ```bash wget -q --timeout=5 google.com -O - ``` It should return: ```text wget: download timed out ``` Access to `google.com` times out because it can resolve DNS but has no egress access to anything other than pods with labels matching `app: nginx` in the `advanced-policy-demo` namespace. # 7. Clean up namespace You can clean up after this tutorial by deleting the advanced policy demo namespace. ```bash kubectl delete ns advanced-policy-demo ``` --- ## Network policy ### Network policy Writing network policies is how you restrict traffic to pods in your Kubernetes cluster. Calico Cloud extends the standard `NetworkPolicy` object to provide advanced network policy features, such as policies that apply to all namespaces. ## Getting started[​](#getting-started) ##### [Policy best practices](https://docs.tigera.io/calico-cloud/network-policy/policy-best-practices) [Learn policy best practices for security, scalability, and performance.](https://docs.tigera.io/calico-cloud/network-policy/policy-best-practices) ##### [Enable a default deny policy for Kubernetes pods](https://docs.tigera.io/calico-cloud/network-policy/beginners/kubernetes-default-deny) [Create a default deny network policy so pods that are missing policy are not allowed traffic until appropriate network policy is defined.](https://docs.tigera.io/calico-cloud/network-policy/beginners/kubernetes-default-deny) ##### [Get started with Calico network policy](https://docs.tigera.io/calico-cloud/network-policy/beginners/calico-network-policy) [Create your first Calico network policies. Shows the rich features using sample policies that extend native Kubernetes network policy.](https://docs.tigera.io/calico-cloud/network-policy/beginners/calico-network-policy) ##### [Get started with network sets](https://docs.tigera.io/calico-cloud/network-policy/networksets) [Learn the power of network sets and why you should create them.](https://docs.tigera.io/calico-cloud/network-policy/networksets) ##### [DNS policy](https://docs.tigera.io/calico-cloud/network-policy/domain-based-policy) [Use domain names to allow traffic to destinations outside of a cluster by their DNS names instead of by their IP addresses.](https://docs.tigera.io/calico-cloud/network-policy/domain-based-policy) ## Policy rules[​](#policy-rules) ##### [Basic rules](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/policy-rules-overview) [Define network connectivity for Calico endpoints using policy rules and label selectors.](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/policy-rules-overview) ##### [Use namespace rules in policy](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/namespace-policy) [Use namespaces and namespace selectors in Calico network policy to group or separate resources. Use network policies to allow or deny traffic to/from pods that belong to specific namespaces.](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/namespace-policy) ##### [Use service rules in policy](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/service-policy) [Use Kubernetes Service names in policy rules.](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/service-policy) ##### [Use service accounts rules in policy](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/service-accounts) [Use Kubernetes service accounts in policies to validate cryptographic identities and/or manage RBAC controlled high-priority rules across teams.](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/service-accounts) ##### [Use external IPs or networks rules in policy](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/external-ips-policy) [Limit egress and ingress traffic using IP address either directly within Calico network policy or managed as Calico network sets.](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/external-ips-policy) ##### [Use ICMP/ping rules in policy](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/icmp-ping) [Control where ICMP/ping is used by creating a Calico network policy to allow and deny ICMP/ping messages for workloads and host endpoints.](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/icmp-ping) ## Policy tiers[​](#policy-tiers) ##### [Get started with policy tiers](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/tiered-policy) [Understand how tiered policy works and supports microsegmentation.](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/tiered-policy) ##### [Change allow-tigera tier behavior](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/allow-tigera) [Understand how to change the behavior of the allow-tigera tier.](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/allow-tigera) ##### [Network policy tutorial](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/policy-tutorial-ui) [Covers the basics of Calico Cloud network policy.](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/policy-tutorial-ui) ##### [Configure RBAC for tiered policies](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/rbac-tiered-policies) [Configure RBAC to control access to policies and tiers.](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/rbac-tiered-policies) ## Policy for services[​](#policy-for-services) ##### [Apply Calico Cloud policy to Kubernetes node ports](https://docs.tigera.io/calico-cloud/network-policy/beginners/services/kubernetes-node-ports) [Restrict access to Kubernetes node ports using Calico Cloud global network policy. Follow the steps to secure the host, the node ports, and the cluster.](https://docs.tigera.io/calico-cloud/network-policy/beginners/services/kubernetes-node-ports) ##### [Apply Calico Cloud policy to services exposed externally as cluster IPs](https://docs.tigera.io/calico-cloud/network-policy/beginners/services/services-cluster-ips) [Expose Kubernetes service cluster IPs over BGP using Calico Cloud, and restrict who can access them using Calico Cloud network policy.](https://docs.tigera.io/calico-cloud/network-policy/beginners/services/services-cluster-ips) ## Policy for extreme traffic[​](#policy-for-extreme-traffic) ##### [Enable extreme high-connection workloads](https://docs.tigera.io/calico-cloud/network-policy/extreme-traffic/high-connection-workloads) [Create a Calico network policy rule to bypass Linux conntrack for traffic to workloads that experience extremely large number of connections.](https://docs.tigera.io/calico-cloud/network-policy/extreme-traffic/high-connection-workloads) ##### [Defend against DoS attacks](https://docs.tigera.io/calico-cloud/network-policy/extreme-traffic/defend-dos-attack) [Define DoS mitigation rules in Calico Cloud policy to quickly drop connections when under attack. Learn how rules use eBPF and XDP, including hardware offload when available.](https://docs.tigera.io/calico-cloud/network-policy/extreme-traffic/defend-dos-attack) ### Policy recommendations ## [📄️ Enable policy recommendations](https://docs.tigera.io/calico-cloud/network-policy/recommendations/policy-recommendations) [Enable continuous policy recommendations to secure unprotected namespaces or workloads.](https://docs.tigera.io/calico-cloud/network-policy/recommendations/policy-recommendations) ## [📄️ Policy recommendations tutorial](https://docs.tigera.io/calico-cloud/network-policy/recommendations/learn-about-policy-recommendations) [Policy recommendations tutorial.](https://docs.tigera.io/calico-cloud/network-policy/recommendations/learn-about-policy-recommendations) ### Enable policy recommendations ## Big picture[​](#big-picture) Use policy recommendations to automatically isolate namespaces with network policy. ## Value[​](#value) One of the best practices for improving the security posture of your Kubernetes cluster is to implement namespace isolation with network policy. Namespace isolation helps to implement a zero-trust and least-privileged security model, where only required communication between namespaces is authorized and everything else is blocked. This helps mitigate the risk of lateral movement within a cluster in the event of an attack. Calico Cloud makes it easy for platform operators to implement namespace isolation without experience in authoring network policy or detailed knowledge of how application workloads are communicating. Calico Cloud analyzes the flow logs that are generated from workloads, and automatically recommends and stages policies for each namespace that can be used for isolation. ## Before you begin[​](#before-you-begin) **Unsupported** - Windows nodes **Required RBAC** To enable/disable and use policy recommendations, you must have the **tigera-network-admin** role or permissions to **update**, **patch**, **get**, **list**, **watch** `projectcalico.org` resources: - tiers - policyrecommendationscopes - stagednetworkpolicies - tier.stagednetworkpolicies - networkpolicies - tier.networkpolicies - globalnetworksets Specifically, you will need access to the `namespace-isolation` tier and to staged and network policies in the `namespace-isolation` tier. **Recommended** Basic knowledge of policies in the web console and tiers: - [Get started with tiered network policy](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/tiered-policy) - [Network policy tutorial](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/policy-tutorial-ui) **Limitations** Creating and managing policy recommendations is available only in the web console. ## How to[​](#how-to) - [Enable policy recommendations](#enable-policy-recommendations) - [Activate and review policy recommendations](#activate-and-review-policy-recommendations) - [Review global settings for workloads](#review-global-settings-for-workloads) - [Update policy recommendations](#update-policy-recommendations) - [Private network recommendations](#private-network-recommendations) - [Troubleshoot policy recommendations](#troubleshoot-policy-recommendations) - [Disable the policy recommendations feature](#disable-the-policy-recommendations-feature) ### Enable policy recommendations **using the web console**[​](#enable-policy-recommendations-using-the-web-console) 1. In the left navbar in the web console, click **Policies**, **Recommendations**. 2. On the opt-in page, click **Enable Policy Recommendations**. The **Policy Recommendations** board is automatically displayed. ![Policy-recommendations-board](https://docs.tigera.io/assets/images/policy-recommendations-board-88cb47f9a55afa186ef56e4caf962794.png) **Notes**: - A policy recommendation is generated for every namespace in your cluster (unless namespaces are filtered out by an Admin using the [selector](https://docs.tigera.io/calico-cloud/reference/resources/policyrecommendations#namespaceSpec#selector) in the PolicyRecommendationScope resource). - Flow logs are continuously monitored for policy recommendations. - Recommended policies are continuously updated until you **Add to policy board** or **Dismiss policy** using the Actions menu. - Policy recommendations are created as **staged network policies** so you can safely observe the traffic before enforcing them. - Traffic originating from the recommended policy's namespace is used to generate egress rules, and traffic destined for the namespace is used to define ingress rules. - To stop policy recommendations from being processed and updated for a namespace, click the **Action** menu, **Dismiss policy**. ### Enable policy recommendations **using kubectl**.[​](#enable-policy-recommendations-using-kubectl) To enable the policy recommendations feature, set the **RecStatus** parameter to `Enabled`, in the [Policy recommendations resource](https://docs.tigera.io/calico-cloud/reference/resources/policyrecommendations). ```bash kubectl patch PolicyRecommendationScope default --type='json' -p='[{"op": "replace", "path": "/spec/namespaceSpec/recStatus", "value": "Enabled"}]' ``` ### Activate and review policy recommendations[​](#activate-and-review-policy-recommendations) Policy recommendations are not enabled until you activate them and move them to the **Active** board. From the Policy Recommendation board, select a policy recommendation (or bulk select) and select, **Add to policy board**. Click on the **Active tab**. You can now view the activated policies in the **Policies Board**. In the left navbar, click **Policies**. Policy recommendations are added to the **namespace-isolation** tier. Note the following: - Staged network policy recommendations work like any other staged network policy. - You cannot move recommended staged policies in the `namespace-isolation` tier. - The name of the `namespace-isolation` tier is fixed and cannot be changed You are now ready to observe traffic flows in Policies board to verify that the policy is authorizing traffic as expected. When a policy works as expected, you can safely enforce it. See [Stage, preview impacts, and enforce policy](https://docs.tigera.io/calico-cloud/network-policy/staged-network-policies) for help. ### Review global settings for workloads[​](#review-global-settings-for-workloads) The default global settings for capturing flows for policy recommendations are based on application workloads with *frequent communication with other namespaces in your cluster*. Global settings are found on the Policy Recommendations board, **Action** menu. ![Global-settings-dialog](https://docs.tigera.io/assets/images/global-settings-35b0ba00eac63feaea850d144857a95c.png) - **Stabilization Period** is the learning time to capture flow logs so that a recommendation accurately reflects the cluster's traffic patterns. - **Processing Interval** is the frequency to process new flow logs and refine recommendations. > **SUCCESS:** For application workloads with less frequent communication, the stabilization period setting may not be long enough to get accurate traffic flows, so you’ll want to increase the time. We recommend that you review your workloads immediately after you enable policy recommendations and adjust the settings accordingly. Changes to all other policy recommendations parameters require Admin permissions and can be changed using the [Policy recommendations resource](https://docs.tigera.io/calico-cloud/reference/resources/policyrecommendations). ### Update policy recommendations[​](#update-policy-recommendations) This section describes common changes you may want to make to policy recommendations. #### Relearn activated recommendations[​](#relearn-activated-recommendations) As new namespace and components are added to a cluster, your activated policy recommendation may need to be updated to reflect those changes. If a policy recommendation has not been enforced, you’ll need to update it to allow traffic. 1. On the **Policies Recommendations** board, click the **Active tab**, which lists the active staged network policies. 2. Select the Actions menu associated with the policy in question, and click **Dismiss policy**. 3. Click the **Dismissed tab**, select the Actions menu, and **Reactivate** the policy. #### Rerun policy recommendations for an enforced policy[​](#rerun-policy-recommendations-for-an-enforced-policy) To generate a new recommendation for an enforced policy, delete the network policy on the **Policy** board. #### Stop policy recommendation updates for a namespace[​](#stop-policy-recommendation-updates-for-a-namespace) 1. On the Policy Recommendations board, click the **Recommendations** tab, which lists the recommendations. 2. Select the recommendation, click the **Actions** menu, and click **Dismiss policy**. To reactivate a policy recommendation for a namespace, select the dismissed staged policy, and from the Actions menu, select **Reactivate**. ### Private network recommendations[​](#private-network-recommendations) If any flow to a private network in your cluster is found, a private rule is automatically created that contains RFC 1918 subnets, which will allow traffic to/from those endpoints. If you need to apply a more restrictive approach, create a [GlobalNetworkSet](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkset) and update it with the desired CIDR blocks. The recommendation engine will identify flows to your private IPs and generate the appropriate NetworkSet Rule. **Notes**: Exclude any CIDR ranges used by the cluster for nodes and pods. ### Troubleshoot policy recommendations[​](#troubleshoot-policy-recommendations) **Problem**: I’m not seeing policy recommendations on the Policy Recommendations board. **Solution/workaround**: Policy recommendations are based on historical flow logs that match a request, and are generated only for flows that have not been addressed by any other policy. As such, there are times when policy recommendations will not be generated: - Not enough traffic history If you recently installed Calico Cloud, you may not have enough traffic history. Workloads must run for some time (around 5 days) to get “typical network traffic” for applications. - Traffic is covered by existing policy Even if your cluster has been running for a long time with traffic, the flows may already be covered by existing policies. To verify why there may not be any recommendations, follow these steps: 1. Go to **Service Graph**, **Default**. 2. Filter flow logs for your namespace. 3. Investigate the content within the `policies` field for the flow logs in question. 4. Validate that no other enforced policy already addresses the flow. **Problem**: Why are egress-to-domain rules being generated for a Kubernetes service? **Solution/workaround**: The policy recommendation controller can only read the cluster domain of the cluster it runs in. If you have managed clusters with a non-default domain (`cluster.local`), the controller will treat egress traffic as though it is to a domain. ### Disable policy recommendations[​](#disable-policy-recommendations) To disable the policy recommendations feature, set the **RecStatus** parameter to `Disabled`, in the [Policy recommendations resource](https://docs.tigera.io/calico-cloud/reference/resources/policyrecommendations). ```bash kubectl patch PolicyRecommendationScope default --type='json' -p='[{"op": "replace", "path": "/spec/namespaceSpec/recStatus", "value": "Disabled"}]' ``` > **SECONDARY:** When disabled, non-active staged policies in the **Policies Recommendations** board are no longer updated. Existing activated and enforced staged network policies are not affected by disabling policy recommendations. ## Additional resources[​](#additional-resources) - [Policy best practices](https://docs.tigera.io/calico-cloud/network-policy/policy-best-practices) ### Policy recommendations tutorial ## Big picture[​](#big-picture) In this tutorial, we show you how recommendations are generated using flow logs in your cluster for traffic to/from namespaces, network sets, private network IPs and public domains. ### Create resources for the tutorial[​](#create-resources-for-the-tutorial) Because the policy recommendation feature requires traffic between endpoints, this step provides these resources for this hands-on tutorial. If your cluster is already generating traffic for policy recommendations, you can skip this step and follow along using your own cluster. 1. Configure felix for fast flow logs collection ```bash kubectl patch felixconfiguration.p default -p '{"spec":{"flowLogsFlushInterval":"10s"}}' ``` 2. Download the [policy recommendation tutorial deployment](https://docs.tigera.io/files/policy-recommendation-deployments.yaml) YAML. 3. Use the following command to create the necessary resources: ```bash kubectl apply -f policy-recommendation-deployments.yaml ``` ### Enable policy recommendation[​](#enable-policy-recommendation) 1. In the web console left navbar, click the **Policies** icon. 2. Select **Recommendations**. 3. Click on **Enable Policy Recommendations**. Wait for the recommendations to be generated. Unless otherwise configured, recommendations will take at least 2m30s to be generated, which is default time for the [Processing Interval](https://docs.tigera.io/calico-cloud/reference/resources/policyrecommendations#spec) setting. Once ready, the recommendations will be listed in the main page, under the **Recommendations** tab. ### Understand the policy recommendation[​](#understand-the-policy-recommendation) You should find a recommendation named `curl-ns` (appended with a five character suffix, like `-vfzgh`) with policy selector: ```text Policy Label selector: [[projectcalico.org/namespace == 'curl-ns']] ``` meaning that this policy pertains to the traffic originating from or destined for the `curl-ns` namespace. The policy will display a list of ingress rules: ```text Allow:Protocol is TCP From: Namespaces [[projectcalico.org/name == 'service-ns']] To:Ports [Port is 80 ] ``` allows ingress traffic, for protocol TCP, on port 80, from the `service-ns` namespace. A list of egress rules: ```text Allow:Protocol is TCP To:Ports [Port is 8080 ] Domains [www.tigera.io] ``` allows egress traffic, for protocol TCP, on port 8080, to domain `www.tigera.io`. ```text Allow:Protocol is TCP To:Ports [Port is 80 ] Namespaces [[projectcalico.org/name == 'service-ns']] ``` allows egress traffic, for protocol TCP, on port 80, to the `service-ns` namespace. ```text Allow:Protocol is UDP To:Ports [Port is 53 ] Namespaces [[projectcalico.org/name == 'kube-system']] ``` allows egress traffic, for protocol UDP, on port 53, to the `kube-system` namespace. ```text Allow:Protocol is TCP To:Ports [Port is 80 ] Endpoints [[projectcalico.org/name == 'public-ips' and projectcalico.org/kind == 'NetworkSet']] Namespaces global() ``` allows egress traffic, for protocol TCP, on port 80, to IPs defined in the global network set named: `public-ips`. ```text Allow:Protocol is TCP To:Ports [Port is 8080 ] Nets [Is 10.0.0.0/8 OR Is 172.16.0.0/12 OR Is 192.168.0.0/16 ] ``` allows egress traffic, for protocol TCP, on port 8080, to private range IPs. ```text Allow:Protocol is TCP To:Ports [Port is 80 ] ``` allows egress traffic, for protocol TCP, on port 80, to public range IPs. ### Investigate the flows that are used to generate the policy rules[​](#investigate-the-flows-that-are-used-to-generate-the-policy-rules) To view flow logs in Service Graph: 1. In the web console left navbar, click **Service Graph**. 2. Select **Default** under the VIEWS option. 3. In the bottom pane you will see flow logs in the Flows tab. To generate rules, the recommendation engine queries for flow logs that are not addressed by any other policy in the cluster. Subsequently, it builds the missing policies necessary for allowing that traffic. ### Understand the flow logs used in policy recommendations[​](#understand-the-flow-logs-used-in-policy-recommendations) To get a better understanding of which flows contributed to generating the rules in your policy, select **Filter Flows** - To find the flows that were used to generate the egress to global network set rule, add: ```text source_namespace = "curl-ns" AND dest_name_aggr = "public-ips" ``` - To find the flows that generated the egress rule to namespace `kube-system`, define query: ```text source_namespace = "curl-ns" AND dest_namespace = "kube-system" ``` You'll notice that each of the flow logs contains a field named, `policies` with a entry like: ```text 1|__PROFILE__|__PROFILE__.kns.curl-ns|allow|0 ``` meaning that the particular flow was not addressed by any other policy within your cluster. You will also find input like: ```text 0|namespace-isolation|curl-ns/namespace-isolation.staged:curl-ns-vfzgh|allow|3 ``` indicating that the 3rd rule defined in policy **curl-ns-vfzgh**, will allow traffic defined by this flow, once the policy is enforced. ### Examine policy traffic[​](#examine-policy-traffic) Examine the **Allowed Bytes** field in the **Recommendations** tab for the `curl-ns-recommendation` policy to get a sense of the total bytes allowed by the policy. Examine the **Allowed/sec** of each rule in the policy to get a sense of the quantity of traffic allowed per second by the rule in question. ### When policy recommendations are not generated[​](#when-policy-recommendations-are-not-generated) You may wonder why you are not getting policy recommendations, even though there is traffic between endpoints. This is because policy recommendations are generated only for flows that are not captured by any other policy in your cluster. To see if policy is already enforcing the traffic in question, search for the flow log in question, examine the `policies` field, and verify that no other enforced policy allows or denies traffic for that flow. ### Policy best practices ## Big picture[​](#big-picture) Policy best practices for run-time security starts with Calico Cloud’s robust network security policy, but other Calico Cloud resources play equally important roles in security, scalability, and performance. Learn Calico Cloud policy best practices and resources that support a zero trust network model: - [Prepare for policy authoring](#prepare-for-policy-authoring) - [Policy best practices for day-one zero trust](#policy-best-practices-for-day-one-zero-trust) - [Policy design for efficiency and performance](#policy-design-for-efficiency-and-performance) - [Policy life cycle tools](#policy-life-cycle-tools) ## Prepare for policy authoring[​](#prepare-for-policy-authoring) ### Determine who can write policy[​](#determine-who-can-write-policy) Any team familiar with deploying microservices in Kubernetes can easily master writing network policies. The challenge in many organizations is deciding who will be given permission to write policy across teams. Although there are different approaches, Calico Cloud policy tools have the flexibility and guardrails to accommodate different approaches. Let’s review two common approaches. - **Microservices teams write policy** In this model, network policy is treated as code, built into and tested during the development process, just like any other critical part of a microservice’s code. The team responsible for developing a microservice has a good understanding of other microservices they consume and depend on, and which microservices consume their microservice. With a defined, standardized approach to policy and label schemas, there is no reason that the teams cannot implement network policies for their microservice as part of the development of the microservice. With visibility in Service Graph, teams can even do basic troubleshooting. - **Dev/Ops writes policy, microservice team focuses on internals** An equally valid approach is to have development teams focus purely on the internals of the microservices they are responsible for, and leave responsibility for operating the microservices with devops teams. A Dev/ops team needs the same understanding as the microservices team above. However, network security may come much later in the organization’s processes, or even as an afterthought on a system already in production. This can be more challenging because getting network policies wrong can have significant production impacts. But using Calico Cloud tools, this approach is still achievable. When you get clarity on who can write policies, you can move to creating tiers. Calico Cloud tiers, along with standard Kubernetes RBAC, provide the infrastructure to meet security concerns across teams. ### Understand the depth of Calico Cloud network policy[​](#understand-the-depth-of-calico-cloud-network-policy) Because Calico Cloud policy goes well beyond the features in Kubernetes policy, we recommend that you have a basic understanding of [network policy and global network policy](https://docs.tigera.io/calico-cloud/network-policy/beginners/calico-network-policy) and how they provide workload access controls. And even though you may not implement the following policies, it is helpful to know the depth of defense that is available in Calico Cloud. - [Policy for services](https://docs.tigera.io/calico-cloud/network-policy/beginners/services/) - [Policy integration for firewalls](https://docs.tigera.io/calico-cloud/network-policy/policy-firewalls/) ### Create policy tiers[​](#create-policy-tiers) **Tiers** are a hierarchical construct used to group policies and enforce higher precedence policies that cannot be circumvented by other teams. As part of your microsegmentation strategy, tiers let you apply identity-based protection to workloads and hosts. Before creating policies, we recommend that you create your tier structure. This often requires internal debates and discussions. As noted previously, Calico Cloud policy workflow has the guardrails you need to allow diverse teams to participate in policy writing. To understand how tiered policy works and best practices, see [Get started with tiered policies](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/tiered-policy). ### Create label standards[​](#create-label-standards) Creating a label standard is often an overlooked step. But if you skip this step, it will cost you in troubleshooting down the road; especially given visibility/troubleshooting is already a challenge in a Kubernetes deployment. **Why are label standards important?** Network policies in Kubernetes depend on **labels and selectors** (not IP addresses and IP ranges) to determine which workloads can talk to each other. As pods dynamically scale up and down, network policy is enforced based on the labels and selectors that you define. So workloads and host endpoints need unique, identifiable labels. If you create duplicate label names, or labels are not intuitive, troubleshooting network policy issues and authoring network policies becomes more difficult. **Recommendations**: - Follow the [Kubernetes guidelines for labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/). If the Kubernetes guidelines do not cover your use cases, we recommend this blog from Tigera Support: [Label standard and best practices for Kubernetes security](https://www.helpnetsecurity.com/2021/05/26/kubernetes-security/). - Develop a comprehensive set of labels that meets the deployment, reporting, and security requirements of different stakeholders in your organization. - Standardize the way you label your pods and write your network policies using a consistent schema or design pattern. - Labels should be defined to achieve a specific and explicit purpose - Use an intuitive language in your label definition that enables a quick and simple identification of labeled Kubernetes objects. - Use label key prefixes and suffixes to identify attributes required for asset classification. - Ensure the right labels are applied to Kubernetes objects by implementing label governance checks in your CI/CD pipeline or at runtime. ### Create network sets[​](#create-network-sets) Network sets and global network sets are grouping mechanisms for arbitrary sets of IPs/subnets/CIDRs or domains. They are key resources for efficient policy design. The key use cases for network sets are: - **Use/reuse in policy to support scaling** You reference network sets in policies using selectors (rather than updating individual policies with CIDRs or domains). - **Visibility to traffic to/from a cluster** For apps that integrate with third-party APIs and SaaS services, you get enhanced visibility to this traffic in Service Graph. - **Global deny lists** Create a “deny-list” of CIDRs for bad actors or embargoed countries in policy. **Recommendation**: Create network sets **and labels** before writing policy. For network set tutorial and best practices, see [Get started with network sets](https://docs.tigera.io/calico-cloud/network-policy/networksets). ## Policy best practices for day-one zero trust[​](#policy-best-practices-for-day-one-zero-trust) ### Create a global default deny policy[​](#create-a-global-default-deny-policy) A global default deny network policy provides an enhanced security posture – so pods without policy (or incorrect policy) are not allowed traffic until appropriate network policy is defined. We recommend creating a global default deny, regardless of whether you use Calico Enterprise and/or Kubernetes network policy. But, be sure to understand the [best practices for creating a default deny policy](https://docs.tigera.io/calico-cloud/network-policy/default-deny) to avoid breaking your cluster. Here are sample [default deny policies](https://docs.tigera.io/calico-cloud/network-policy/beginners/kubernetes-default-deny). ### Define both ingress and egress network policy rules for every pod in the cluster[​](#define-both-ingress-and-egress-network-policy-rules-for-every-pod-in-the-cluster) Although defining network policy for traffic external to clusters (north-south) is certainly important, it is equally important to defend against attacks for east-west traffic. Simply put, **every connection from/to every pod in every cluster should be protected**. Although having both doesn’t guarantee protection against other attacks and vulnerabilities, one innocuous workload can lead to exposure of your most critical workloads. For examples, see [basic ingress and egress policies](https://docs.tigera.io/calico-cloud/network-policy/beginners/calico-network-policy). ## Policy design for efficiency and performance[​](#policy-design-for-efficiency-and-performance) Teams can write policies that work, but ultimately you want policies that also scale, and do not negatively impact performance. If you follow a few simple guidelines, you’ll be well on your way to writing efficient policy. ### Use global network policy only when all rules apply globally[​](#use-global-network-policy-only-when-all-rules-apply-globally) - **Do** Use global network policy for cluster-wide scope when all rules apply to multiple namespaces or host endpoints. For example, use a global network policy to create a deny-list of CIDRs for embargoed countries, or for global default deny everywhere, even for new namespaces. Why? Although at the level of packet processing there is no difference between network policy and global network, for CPU usage, one global network policy is faster than a large number of network policies. - **Avoid** Using a global network policy as a way to combine diverse, namespaced endpoints with different connectivity requirements. Although creating such a policy can work, appears efficient and is easier to view than several separate network policies, it is inefficient and should be avoided. Why? Putting a lot of anything in policy (rules, CIDRs, ports) that are manipulated by selectors is inefficient. iptables/eBPF rules depend on minimizing executions and updates. When a selector is encountered in a policy rule, it is converted into one iptables rule that matches on an IP set. Then, different code keeps the IP sets up to date; this is more efficient than updating iptables rules. Also, because iptables rules execute sequentially in order, having many rules results in longer network latencies for the first packet in a flow (approximately 0.25-0.5us per rule). Finally, having more rules slows down programming of the data plane, making policy updates take longer. **Example: Inefficient global network policy** The following policy is a global network policy for a microservice that limits all egress communication external to the cluster in the security tier. Does this policy work? Yes. And logically, it seems to cleanly implement application controls. ```yaml 1 apiVersion: projectcalico.org/v3 2 kind: GlobalNetworkPolicy 3 metadata: 4 name: security.allow-egress-from-pods 5 spec: 6 tier: security 7 order: 1 8 selector: all() 9 egress: 10 - action: Deny 11 source: 12 namespaceSelector: projectcalico.org/namespace starts with "tigera" 13 destination: 14 selector: threatfeed == "feodo" 15 - action: Allow 16 protocol: TCP 17 source: 18 namespaceSelector: projectcalico.org/name == "sso" 19 ports: 20 - '443' 21 - '80' 22 destination: 23 domains: 24 - '*.googleapis.com' 25 - action: Allow 26 protocol: TCP 27 source: 28 selector: psql == "external" 29 destination: 30 ports: 31 - '5432' 32 domains: 33 - '*.postgres.database.azure.com' 34 - action: Allow 35 protocol: TCP 36 source: {} 37 destination: 38 ports: 39 - '443' 40 - '80' 41 domains: 42 - '*.logic.azure.com' 43 - action: Allow 44 protocol: TCP 45 source: {} 46 destination: 47 ports: 48 - '443' 49 - '80' 50 domains: 51 - '*.azurewebsites.windows.net' 52 - action: Allow 53 protocol: TCP 54 source: 55 selector: 'app in { "call-archives-api" }||app in { "finwise" }' 56 destination: 57 domains: 58 - '*.documents.azure.com' 59 - action: Allow 60 protocol: TCP 61 source: 62 namespaceSelector: projectcalico.org/name == "warehouse" 63 destination: 64 ports: 65 - '1433' 66 domains: 67 - '*.database.windows.net' 68 - action: Allow 69 protocol: TCP 70 source: {} 71 destination: 72 nets: 73 - 65.132.216.26/32 74. - 10.10.10.1/32 75 ports: 76 - '80' 77 - '443' 78 - action: Allow 79 protocol: TCP 80 source: 81 selector: app == "api-caller" 82 destination: 83 ports: 84 - '80' 85 - '443' 86 domains: 87 - api.example.com 88 - action: Allow 89 source: 90 selector: component == "tunnel" 91 - action: Allow 92 destination: 93 selector: all() 94 namespaceSelector: all() 95 - action: Deny 96 types: 97 - Egress ``` **Why this policy is inefficient** First, the policy does not follow guidance on use for global network policy: that all rules apply to the endpoints. So the main issue is inefficiency, although the policy works. The main selector `all()` (line 8) means the policy will be rendered on every endpoint (workload and host endpoints). The selectors in each rule (for example, lines 12 and 14) control traffic that are matched by that rule. So, even if the host doesn’t have any workloads that match `"selector: app == "api-caller"`, you’ll still get the iptables/eBPF rule rendered on every host to implement that rule. If this sample policy had 100 pods, that’s a 10 - 100x increase in the number of rules (depending on how many local endpoints match each rule). In short, it adds: - Memory and CPU to keep track of all the extra rules - Complexity to handle changes to endpoint labels, and to re-render all the policies too. ### Avoid policies that may select unwanted endpoints[​](#avoid-policies-that-may-select-unwanted-endpoints) The following policy is for an application in a single namespace, `app1-ns` namespace. There are two microservices that are all labeled appropriately: - microservice 1 has `app: app1`, `svc: svc1` - microservice 2 has `app: app1`, `svc: svc2` The following policy works correctly and does not incur a huge performance hit. But it could select additional endpoints that were not intended. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: application.app1 namespace: app1-ns spec: tier: application order: 10 selector: app == "app1" types: - Ingress ingress: - action: Allow source: selector: trusted-ip == "load-balancer" destination: selector: svc == "svc1" ports: - 10001 protocol: TCP - action: Allow source: selector: svc == "svc1" destination: selector: svc == "svc2" ports: - 10002 protocol: TCP ``` The policy incorrectly assumes that the main policy selector (`app == "app1"`) will be combined (ANDed) with the endpoint selector, and only for certain policy types. In this case, - **Ingress** - combines policy selector and *destination endpoint selector* or - **Egress** - combines policy selector and *source endpoint selector* But if the assumptions behind the labels are not understood by other policy authors and are not correctly assigned, the endpoint selector may select *additional endpoints that were not intended*. For ingress policy, this can open up the endpoint to more IP addresses than necessary. This unintended consequence would be exacerbated if the author used a global network policy. ### Put multiple relevant policy rules together in the same policy[​](#put-multiple-relevant-policy-rules-together-in-the-same-policy) As discussed previously, it is better to create separate policies for different endpoint connectivity rules, than a single global network policy. However, you may interpret this to mean that the best practice is to make unique policies that do not aggregate any rules. But that is not the case. Why? When Calico Cloud calculates and enforces policy, it updates the iptables/eBPF and reads policy changes and pod/workload endpoints from the datastore. The more policies in memory, the more work it takes determine which policies match a particular endpoint. If you group more rules into one policy, there are fewer policies to match against. ### Understand effective use of label selectors[​](#understand-effective-use-of-label-selectors) Label selectors abstract network policy from the network. Misuse of selectors can slow things down. As discussed previously, the more selectors you create, the harder Calico Cloud works to find matches. The following policy shows an inefficient use of selectors. Using `selector: all()` renders the policy on all nodes for all workloads. If there are 10,000 workloads, but only 10 match label==foo, that is very inefficient at the data plane level. ```yaml selector: all() ingress: - source: selector: label == 'bar' destination: selector: label == 'foo' ``` The best practice policy below allows the same traffic, but is more efficient and scalable. Why? Because the policy will be rendered only on nodes with workloads that match the selector `label==foo`. ```yaml selector: label == 'foo' ingress: source: selector: label == 'bar' ``` Another common mistake is using `selector: all()` when you don’t need to. `all()` means *all workloads* so that will be a large IP set. Whenever there's a source/destination selector in a rule, it is rendered as an IP set in the data plane. ```yaml source: selector: all() ``` ### Put domains and CIDRs in network sets rather than policy[​](#put-domains-and-cidrs-in-network-sets-rather-than-policy) Network sets allow you to specify CIDRs and/or domains. As noted in [Network set best practices](https://docs.tigera.io/calico-cloud/network-policy/policy-best-practices), we do not recommend putting large CIDRs or domains directly in policy. Although nothing stops you from do this in policy, using network sets is more efficient and supports scaling. ## Policy life cycle tools[​](#policy-life-cycle-tools) ### Preview, stage, deploy[​](#preview-stage-deploy) A big obstacle to adopting Kubernetes is not having confidence that you can effectively prevent, detect, and mitigate across diverse teams. The following policy life cycle tools in the web console (**Policies** tab) can help. - **Policy recommendations** Get a policy recommendation for unprotected workloads. Speeds up learning, while supporting zero trust. - **Policy impact preview** Preview the impacts of policy changes before you apply them to avoid unintentionally exposing or blocking other network traffic. - **Policy staging and audit modes** Stage network policy so you can monitor traffic impact of both Kubernetes and Calico Cloud policy as if it were actually enforced, but without changing traffic flow. This minimizes misconfiguration and potential network disruption. For details, see [Policy life cycle tools](https://docs.tigera.io/calico-cloud/network-policy/staged-network-policies). ### Do not trust anything[​](#do-not-trust-anything) Zero trust means that you do not trust anyone or anything. Calico Cloud handles authentication on a per request basis. Every action is either authorized or restricted, and the default is everything is restricted. To apply zero trust to policy and reduce your attack surface and risk, we recommend the following: - Ensure that all expected and allowed network flows are explicitly allowed; any connection not explicitly allowed is denied - Create a quarantine policy that denies all traffic that you can quickly apply to workloads when you detect suspicious activity or threats ## Additional resources[​](#additional-resources) - [Troubleshoot policies](https://docs.tigera.io/calico-cloud/network-policy/policy-troubleshooting) - [Security and policy best practices blog](https://www.tigera.io/blog/kubernetes-security-policy-10-critical-best-practices/) ### Policy tiers ## [📄️ Get started with policy tiers](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/tiered-policy) [Understand how tiered policy works and supports microsegmentation.](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/tiered-policy) ## [📄️ Network policy tutorial](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/policy-tutorial-ui) [Covers the basics of Calico Cloud network policy.](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/policy-tutorial-ui) ## [📄️ Change allow-tigera tier behavior](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/allow-tigera) [Understand how to change the behavior of the allow-tigera tier.](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/allow-tigera) ## [📄️ Configure RBAC for tiered policies](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/rbac-tiered-policies) [Configure RBAC to control access to policies and tiers.](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/rbac-tiered-policies) ### Get started with policy tiers ## Seamless network policy integration[​](#seamless-network-policy-integration) **Network policy** is the primary tool for securing a Kubernetes network. It lets you restrict network traffic in your cluster so only the traffic that you want to flow is allowed. Calico Cloud provides more robust policy than Kubernetes, but you can use them together -- seamlessly. Calico Cloud supports: - Calico Cloud network policy, (namespaced) - Calico Cloud global network policy (non-namespaced, global) - Kubernetes network policy ## Tiers: what and why?[​](#tiers-what-and-why) **Tiers** are a hierarchical construct used to group policies and enforce higher precedence policies that cannot be circumvented by other teams. As you will learn in this tutorial, tiers have built-in features that support workload microsegmentation. All Calico Cloud and Kubernetes network policies reside in tiers. You can start "thinking in tiers" by grouping your teams and types of policies within each group. For example, we recommend these three tiers (platform, security, and application). ![policy-types](https://docs.tigera.io/assets/images/policy-types-474301e8cc7ae4fd70228fb5259a39cc.png) Next, you can determine the priority of policies in tiers (from top to bottom). In the following example, the platform and security tiers use Calico Cloud global network policies that apply to all pods, while developer teams can safely manage pods within namespaces using Kubernetes network policy for their applications and microservices. ![policy-tiers](https://docs.tigera.io/assets/images/policy-tiers-b66ba4e79aa0ce4cdbea2fed2095e53d.png) ## Create a tier and policy[​](#create-a-tier-and-policy) To create a tier and policy in the web console: 1. In the left navbar, click **Policies**. 2. On the **Policies Board**, click **Add Tier**. 3. Name the tier, select **Order, Add after** `tigera-security`, and save. 4. To create a policy in the tier, click **+ Add policy**. You can export all policies or a single policy to a YAML file. Here is a sample YAML that creates a security tier and uses `kubectl` to apply it. ```yaml apiVersion: projectcalico.org/v3 kind: Tier metadata: name: security spec: order: 300 ``` ```bash kubectl apply -f security.yaml ``` ## The default tier:[​](#the-default-tier) The default tier is created during installation and has the order of 1,000,000. This value is fixed, and cannot be changed. The desire is to evaluate policies in the default tier as last. ![default-tier](https://docs.tigera.io/assets/images/default-tier-10ac5a5bbb60955d2db4267e739f79b2.png) The default tier is where: - You manage all Kubernetes network policies - Network and global network policies are placed when you upgrade from Project Calico to Calico Cloud - Recommended policies are placed when you use the **Recommend a policy** feature ## The admin network policy tier:[​](#the-admin-network-policy-tier) The `Tier` with name `adminnetworkpolicy` is where all [Kubernetes admin network policies](https://network-policy-api.sigs.k8s.io/reference/examples/) reside. It is automatically created during installation and has the order of 1,000 and a default action of `Pass`. This is fixed, and should not be changed. In addition to `AdminNetworkPolicy` resources, you can also add Calico Cloud `NetworkPolicy`/`GlobalNetworkPolicy` resources to the admin network policy `Tier`. The `AdminNetworkPolicy` `priority` field maps to the `order` field in the Calico Cloud `NetworkPolicy`/`GlobalNetworkPolicy` resources so that the two types of policy can be ordered together. ## The baseline admin network policy tier:[​](#the-baseline-admin-network-policy-tier) The `Tier` with name `baselineadminnetworkpolicy` is where the singleton [Kubernetes baseline admin network policy](https://network-policy-api.sigs.k8s.io/reference/examples/) resides. It is automatically created during installation and has the order of 10,000,000 and a default action of `Pass`. This is fixed, and should not be changed. In addition to the `BaselineAdminNetworkPolicy` resource, you can also add Calico Cloud `NetworkPolicy`/`GlobalNetworkPolicy` resources to the baseline admin network policy `Tier`. The `BaselineAdminNetworkPolicy` is treated as if it had an `order` of 1,000, allowing Calico Cloud `NetworkPolicy`/`GlobalNetworkPolicy` resources to be inserted with higher/lower precedence than the `BaselineAdminNetworkPolicy`. ## System tiers[​](#system-tiers) System tiers are added during installation and are hidden by default. - **allow-tigera** tier contains policies to secure Calico Cloud components and are controlled by the Tigera Operator. These policies should not be edited, and the tier should not be moved. Inadvertent changes are automatically reverted by the Operator to ensure your cluster is always protected. > **WARNING:** Although it is possible to change the behavior of the `allow-tigera` using adjacent tiers, it is not a trivial task. You can break critical cluster traffic and impact the operation of Calico Cloud. To prevent loss of cluster services, see [Change allow-tigera tier behavior](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/allow-tigera), and contact Support for help. ## Moving tiers[​](#moving-tiers) You can move tiers by dragging and moving them in the graphical sequence, but all tiers must be visible first before you reorder tiers. To show all tiers, click **View** and select all of the tiers in the Show tiers list. ![hidden-tiers](https://docs.tigera.io/assets/images/moving-tiers-a7ec6a1e2ad1fb441fd203f196f832a6.png) Now you can reorder tiers by dragging and moving them. ## Tier order[​](#tier-order) Tiers are sorted by their orders (from left to right), starting from the lowest order (the highest priority) tiers. ![tier-order](https://docs.tigera.io/assets/images/tier-order-76f0aa060f5a535aa811390ea964f30f.png) In the example above, tier priorities are as follows: - **security tier** - is higher priority than platform tier - **platform tier** - is higher priority than default tier - **default tier** - is the lowest priority The tier you put as the highest priority (after system tiers), depends on your environment. In compliance-driven environments, the security tier may be the highest priority (as shown above). There is no one-size-fits-all order. ## Policy processing[​](#policy-processing) Policies are processed in sequential order from lowest order (highest priority) to highest order (lowest priority), which is from top to bottom in the following diagram. ![policy-processing](https://docs.tigera.io/assets/images/policy-processing-4884083e5b9e9b0b645b1e4257aa058e.png) Two mechanisms drive how traffic is processed across tiered policies: - Labels and selectors - Policy action rules It is important to understand the roles they play. ### Labels and selectors[​](#labels-and-selectors) Instead of IP addresses and IP ranges, network policies in Kubernetes depend on labels and selectors to determine which workloads can talk to each other. Workload identity is the same for Kubernetes and Calico Cloud network policies: as pods dynamically come and go, network policy is enforced based on the labels and selectors that you define. The following diagrams show the relationship between all of the elements that affect traffic flow: - **Tiers** group and order policies - **Policy action rules** define how to process traffic in and across tiers, and policy labels and selectors specify how groups of pods are allowed to communicate with each other and other network endpoints - The **CNI**, **Calico Cloud components**, and underlying **dataplane** (iptables/eBPF) all make use of labels and selectors as part of routing traffic. ![tier-funnel](https://docs.tigera.io/assets/images/tier-funnel-54307b875cf8092dc31db531fdb328db.png) ### Policy action rules[​](#policy-action-rules) Calico Cloud network policy uses action rules to specify how to process traffic/packets: - **Allow or Deny** - traffic is allowed or denied and the packet is handled accordingly. No further rules are processed. - **Pass** - skips to the next tier that contains a policy that applies to the endpoint, and processes the packet. If the tier applies to the endpoint but no action is taken on the packet, the packet is dropped. - **Log** - creates a log, and evaluation continues processing to the next rule ### Implicit default deny[​](#implicit-default-deny) As shown in the following diagram, at the end of each tier is an implicit default deny. This is a safeguard that helps mitigate against unsecured policy. Because of this safeguard, you must either explicitly update tier's default action to **Pass** or apply a Pass action rule when you want traffic evaluation to continue. In the following example, the Pass action in a policy ensures that traffic evaluation continues, and overrides the implicit default deny. ![implicit-deny](https://docs.tigera.io/assets/images/implicit-deny-bb4e0bb76a63f4c41e3ee7885ff1362f.svg) Let’s look at a Dev/Ops global network policy in a high precedence tier (Platform). The policy denies ingress and egress traffic to workloads that match selector, `env == "stage"`. To ensure that policies continue to evaluate traffic after this policy, the policy adds a Pass action for both ingress and egress. **Pass action rule example** ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: devops.stage-env spec: tier: devops order: 255 selector: env == "stage" ingress: - action: Deny source: selector: env != "stage" - action: Pass egress: - action: Deny destination: selector: env != "stage" - action: Pass types: - Ingress - Egress ``` The other option, is to update tier's default action to **Pass** to override the implicit default deny. **Pass default action tier example** ```yaml apiVersion: projectcalico.org/v3 kind: Tier metadata: name: devops spec: order: 300 defaultAction: Pass ``` ### Policy endpoint matching across tiers[​](#policy-endpoint-matching-across-tiers) Whoever is responsible for tier creation, also needs to understand how policy selects matching endpoints across tiers. For normal policy processing (without apply-on-forward, pre-DNAT, and do-not-track), if no policies within a tier apply to endpoints, the tier is skipped, and the tier's implicit deny behavior is not executed. In the following example, **policy D** in the Security tier includes a Pass action rule because we want traffic evaluation to continue to the next tier in sequence. In the Platform tier, there are no selectors in policies that match endpoints so the tier is skipped, including the end of tier deny. Evaluation continues to the Application tier. **Policy J** is the first policy with a matching endpoint. ![endpoint-match](https://docs.tigera.io/assets/images/endpoint-match-ecaab9647b9615e9154c5e134a8cba8e.svg) ### Default endpoint behavior[​](#default-endpoint-behavior) Also, tier managers need to understand the default behavior for endpoints based on whether the endpoint is known or unknown, and the endpoint type. As shown in the following table: - **Known endpoints** - Calico Cloud resources that are managed by Felix - **Unknown endpoints** - interfaces/resources not recognizable as part of our data model | Endpoint type | Default behavior for known endpoints | Default behavior for unknown endpoints (outside of our data model) | | ---------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------ | | Workload, Calico Cloud | Deny | Deny | | Workload, Kubernetes | Allow ingress from same Kubernetes namespace; allow all egress | Deny | | Host | Deny. With exception of auto host endpoints, which get `default-allow`. | Fall through and use iptables rules | ## Best practices for tiered policy[​](#best-practices-for-tiered-policy) To control and authorize access to Calico Cloud tiers, policies, and Kubernetes network policies, you use Kubernetes RBAC. Security teams can prevent unauthorized viewing or modification of higher precedence (lower order) tiers, while still allowing developers or service owners to manage the detailed policies related to their workloads. We recommend: - Limit tier creation permissions to Admin users only; creating and reordering tiers affects your policy processing workflow - Limit full CRUD operations on tiers and policy management to select Admin users - Review your policy processing whenever you add/reorder tiers For example, you may need to update Pass action rules to policies before or after the new tier. Intervening tiers may require changes to policies before and after, depending on the endpoints. - Use the **policy preview** feature to see effects of policy in action before enforcing it, and use the **staged network policy** feature to test the entire tier workflow before pushing it to production ## Additional resources[​](#additional-resources) - For details on using RBAC for fine-grained access to tiers and policies, see [Configure RBAC for tiered policies](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/rbac-tiered-policies). ### Network policy tutorial ## What you will learn:[​](#what-you-will-learn) - How to create a policy in the web console - How labels and selectors work - Basics of policy ordering and tiers ## Scenario[​](#scenario) Let's start with a sample Kubernetes cluster. ![policy-tutorial-overview](https://docs.tigera.io/assets/images/policy-tutorial-overview-91920c6a1aa65855cfb92bf0cbc3eac6.png) | Item | Description | | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Kubernetes cluster | A Kubernetes cluster with four namespaces and three nodes that run the pods in the cluster. | | Namespace | Four namespaces named blue, red, green, and purple represent different applications running in the cluster. | | Pod | Pods with meaningful labels for our applications: - FE (frontend pods) - BE (backend pods) | | NetworkSet | An arbitrary set of IP subnetworks/CIDRs or domains that can be matched by standard label selectors in Kubernetes or Calico Cloud network policy. Network sets are a Calico Cloud namespaced resource. | | GlobalNetwork Set | An arbitrary set of IP subnetworks/CIDRs or domains that can be matched by standard label selectors in Kubernetes or Calico Cloud network policy. Global network sets are a Calico Cloud global resource. | | ServiceAccount | Provides an identity for processes that run in a pod. Service accounts are a Kubernetes namespaced resource. | | HostEndpoint (HEP) | Physical or virtual interfaces attached to a host that runs Calico Cloud. HEPs enforce Calico Cloud policy on the traffic that enters or leaves the host’s default network namespace through the interfaces. HEPs are a Calico Cloud global resource. | | External component | A machine (physical or virtual) that runs outside of the Kubernetes cluster. | ## Create a network policy[​](#create-a-network-policy) To follow along in the web console, click **Policies**. There are three main parts to every Calico Cloud policy: - **Scope** - namespace or global - **Applies to** - objects within the above scope to which policy rules will be applied using labels and selectors - **Type** - whether this policy affects ingress, egress, or both - Ingress - policy rules to apply to connections inbound to the selected objects - Egress - policy rules to apply to connections outbound from the selected objects ![policy-parts](https://docs.tigera.io/assets/images/policy-parts-a4bf9b6715c64f203b25d98edf01b973.png) Let's look at each part. ## Scope[​](#scope) Scope defines the reach of your policy. Use this dropdown to determine whether your policy applies globally or to a specific namespace. Think of scope as the "top-level scope" that can be further specified using the "Applies to" selection that follows. - **Global** If you select global, but do not add entries in the **Applies to** field to further limit scope, *every pod and host endpoint (HEP) in our cluster would be in scope*. The following example uses the global option to limit the scope to all pods and HEPs (noted by check marks). ![policy-tutorial-global-scope](https://docs.tigera.io/assets/images/policy-tutorial-global-scope-24e80fa651ccf2a5666ccfba71ef725a.png) - **Namespace** If you select namespace, but do not add entries in the **Applies to** field to further limit scope, *every pod in this policy's namespace would be in scope*. The following example uses the namespace option to limit the scope to pods in the RED namespace. ![policy-tutorial-namespace-scope](https://docs.tigera.io/assets/images/policy-tutorial-namespace-scope-433e092c2e089cde8c84fdeecfda1eb5.png) ### Applies to[​](#applies-to) As discussed above, selecting **Applies to** lets you further limit pods in a policy. You can think of it as the "top-level endpoint selector". You define labels on your endpoints, namespaces, and service accounts, then use label selectors to limit connections by matching the following object types: - **Endpoints** Specify one or more label selectors to match specific endpoints, or select all endpoints - **Namespaces** (available only when the Scope is global) Specify one or more label selectors to match specific namespaces, or select all namespaces - **Service Accounts** Specify or more label selectors to match specific service accounts, or select all service accounts For example, if we select the BLUE namespace and apply it to only pods with the label, `app/tier == FE`, ![blue-namespace](https://docs.tigera.io/assets/images/blue-namespace-86cd99e724dc5d0e8ee70119c4a55899.png) the resulting scope in our diagram would be only the pods labeled, `FE`: ![blue-namespace-pods](https://docs.tigera.io/assets/images/blue-namespace-pods-3ee6e89b3aff34001f47c8fe6462245a.png) ### Type[​](#type) In the Type section, you specify whether the policy impacts ingress, egress, or both. Note that ingress and egress are defined from the point of view of the *scoped endpoints* (pods or host endpoints). In the previous diagram, the scoped endpoints are the three pods labeled, `app/tier:FE`. - Ingress rules filter traffic *coming to* the scoped endpoints - Egress rules filter traffic *leaving* the scoped endpoints Select the **Ingress** rule, and click **+ Add ingress rule** to access the **Create New Policy rules** page. ### Endpoint selector[​](#endpoint-selector) The endpoint selector lets you select the endpoint traffic that is matched within the scope you've defined in the policy. In our example, the policy is scoped to endpoints that have the `app/tier == FE` label in the BLUE namespace. In the context of an egress rule, when we add the `app/tier == BE` endpoint selector, all TCP traffic from endpoints that have the`app/tier == BE` label will be allowed to the `app/tier == FE` endpoints. ![policy-tutorial-endpoint-selector](https://docs.tigera.io/assets/images/policy-tutorial-endpoint-selector-662a22e316bdcccc36f86a372b6c7cac.png) Note that endpoints that have the `app/tier == BE` label in other namespaces are not matched because the policy is namespace scoped. ### Namespace selector[​](#namespace-selector) This is where things can get interesting. In the previous example, we did not select anything in the namespace selector. Let's change the namespace selector to have both the BLUE and GREEN namespaces. ![endpoint-selector-blue-green](https://docs.tigera.io/assets/images/endpoint-selector-blue-green-7194f4576aba9e89d459bf959138ab7b.png) Although the overall policy is scoped to the BLUE namespace, we can match endpoints in other namespaces on a per-rule basis. Note that the top-level scope that you select remains unchanged, meaning that the policy is still applied only to endpoints in the BLUE namespace. ![namespace-selector](https://docs.tigera.io/assets/images/namespace-selector-8d78dbecf512a4ba4d5cc8c68c750211.png) ### Network selector[​](#network-selector) Using the Nets selector, we can add CIDR addresses to be matched by the policy rule. ![network-selector](https://docs.tigera.io/assets/images/network-selector-79b67d3c2be0d99306e4d71daed1392d.png) ### Service account selector[​](#service-account-selector) Network policies can be also applied to the endpoint’s service account. Using the service account selector, we can apply rules to traffic from any endpoint whose service account matches the name or label selector. ![service-account-selector](https://docs.tigera.io/assets/images/service-account-selector-bf11d73497eee691ff6a4f576c805009.png) ### Use Match All for wider matches in policy rules[​](#use-match-all-for-wider-matches-in-policy-rules) The **Match All** policy rule (`all()` in YAML) matches traffic for: - All endpoints in a namespace (if the policy scope is namespace) - All endpoints (if the policy scope is global) Let's look at an example of using **Match All** traffic in a namespaced policy: - Scope is namespaced (BLUE) - Applies to `app/tier == FE` Suppose we want to match traffic to the pod labeled `BE`, and the Calico Cloud `networkset-1`. ![match-all-namespace](https://docs.tigera.io/assets/images/match-all-namespace-010e361b3f0ee6da609365d075d9bc1b.png) To do this, we can use the policy rule endpoint selector, **Match All**. ![match-all-endpoints](https://docs.tigera.io/assets/images/match-all-endpoints-639cb26986109bbe6968c87638c89704.png) Not only is the pod labeled `BE` included, but also the Calico Cloud `networkset-1`. ![match-all-endpoints-example](https://docs.tigera.io/assets/images/match-all-endpoints-example-62994655a94219130839493e17ca5b72.png) Note that we could have created individual selectors to match pods labeled, `BE` and for the `network-set-1`. **Match All traffic with namespace selectors** In the following example, if we select **Match All** endpoints, but in the **Namespace selector**, we select both the BLUE and GREEN namespaces, the results for matching are: all pods and network sets in the BLUE and GREEN namespaces. ![namespace-match-all](https://docs.tigera.io/assets/images/namespace-match-all-aa7e7d8370762e1626e17d3278b4870c.png) **Global selector** Let's see what happens when we select the **Global** selector. ![namespace-selector-global](https://docs.tigera.io/assets/images/namespace-selector-global-81caeb9b9bdbd7d5c1d917162efcc1dc.png) In our example, the Global selector selects HEPs and global network sets are selected. You might think that Global (`global()` in YAML) would select all endpoints, but it doesn't. Global means "do not select any namespaced resources" (which includes namespaced network set resources). Another way to express it is, do not select any workload endpoints. ![heps-networksets](https://docs.tigera.io/assets/images/heps-networksets-f743844e0eb563099dc2d5e4ce429ff2.png) **Endpoint selector, unspecified** Next, let's see what happens when the policy rule does not specify any selection criteria. In this example, the rule selects all workloads, network sets, endpoints, and host endpoints within scope of the policy, including external components (the VM database). ![unspecified](https://docs.tigera.io/assets/images/unspecified-12bfc7761304972a82799bc47488be8c.png) Now that you know the basic elements of a network policy, let's move on to policy ordering and tiers. ## Policy ordering[​](#policy-ordering) Calico Cloud policies can have order values that control the order of precedence. For both network policies and global network policies, Calico Cloud applies the policy with the lowest value first. ![policy-ordering](https://docs.tigera.io/assets/images/policy-ordering-6da9cd2fc49687c4fde98872dee2a52e.png) ### Mixing Kubernetes and Calico Cloud policies[​](#mixing-kubernetes-and-calico-cloud-policies) Kubernetes and Calico Cloud policies work side by side without a problem. However, Kubernetes network policies cannot assign an order value, so Calico Cloud will set an implicit order value of 1000 to any Kubernetes network policies. > **SECONDARY:** Policies are immediately applied to any new connections. However, for existing connections that are already open, the policy changes will only take effect after the connection has been reestablished. This means that any ongoing sessions may not immediately reflect policy changes until they are initiated again. ### Calico Cloud policies with no order value[​](#calico-cloud-policies-with-no-order-value) Calico Cloud policies with order values take precedence. Policies without order values take lesser precedence and are processed alphabetically. ## Tiers[​](#tiers) Tiers are a hierarchical construct used to group policies and enforce higher precedence policies that cannot be circumvented by other teams. Access to tiers is controlled using user role permissions. For example, a security team can implement high-level policy (for example, blocking access to/from IP ranges in particular countries), while developers in a later tier can control specific rules for the microservices of an app running in the cluster. ### Policy processing overview[​](#policy-processing-overview) When a new connection is processed by Calico Cloud, each tier that contains a policy that selects the endpoint processes the connection. Tiers are sorted by their order - the smallest number first. Policies in each tier are then processed in order from lowest to highest. For example, a policy of 800 is ordered before a policy of order 1000. - If a network policy or global network policy in the tier allows or denies the connection, then evaluation is done: the connection is handled accordingly. - If a network policy or global network policy in the tier passes the connection, the next tier containing a policy that selects the endpoint processes the connection After a Pass action, if no subsequent tier contains any policies that apply to the pod, the connection is allowed. If the tier contains policies that apply to the endpoint, but the policies take no action on the connection, the connection is dropped by an implicit deny. If no tiers contain policies that apply to the endpoint, the connection is allowed by an implicit allow. ### Policies with no order value[​](#policies-with-no-order-value) You can create policies without an order value. When a policy with no order value is placed in a tier with other policies that do have an order value, the policies are processed as follows: - Policies are evaluated from smallest to largest order value within the tier - Policies with no order value are processed last in the tier, but before the implicit deny - When multiple policies without an order value are present in a tier, they are processed in alphabetical order. However, we do not recommended relying on alphabetical ordering because it hard to operationalize. ### How policy action rules affect traffic processing[​](#how-policy-action-rules-affect-traffic-processing) It is also important to understand that Calico Cloud policy action rules affect how traffic and connections are processed. Let's go back to the drop-down menu on the Create New Policy Rule page. Action defines what should happen when a connection matches this rule. ![policy-tutorial-action](https://docs.tigera.io/assets/images/policy-tutorial-action-aad95b91d186a2053e485ec3ec229ade.png) - **Allow or Deny** - traffic is allowed or denied and the connection is handled accordingly. No further rules are processed. - **Pass** - skips to the next tier that contains a policy that applies to the endpoint, and processes the connection. If the tier applies to the endpoint but no action is taken on the connection, the connection is dropped. - **Log** - creates a log, and evaluation continues processing to the next rule ## Additional resources[​](#additional-resources) The following topics go into further detail about concepts described in this tutorial: - [Get started with network policy](https://docs.tigera.io/calico-cloud/network-policy/beginners/calico-network-policy) - [Service account selectors](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/service-accounts) - [Get started with tiered network policy](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/tiered-policy) - [Get started with network sets](https://docs.tigera.io/calico-cloud/network-policy/networksets) ### Change allow-tigera tier behavior > **WARNING:** The `allow-tigera` tier contains policies that secure Calico Cloud components and is critical to cluster integrity. It is controlled by the Tigera Operator, and policies in the tier should not be edited, and the tier should not be moved. Although you can change the behavior of allow-tigera using adjacent tiers, you can inadvertently break critical cluster traffic. We highly recommend that you work with Support to implement changes around `allow-tigera` to prevent service disruption. ## Big picture[​](#big-picture) Change traffic behavior of the tier that secures Calico Cloud components. ## Value[​](#value) Although the tier that secures Calico Cloud components cannot be changed, you can create policies in adjacent tiers to change its behavior. ## Concepts[​](#concepts) Calico Cloud automatically creates the `allow-tigera` tier during installation with network policies that select traffic to and from Tigera components. These policies ensure that traffic required for Calico Cloud operation is allowed, and that any unnecessary traffic involving Tigera components is denied. This tier prevents disruption of Calico Cloud functionality in case of network policy misconfiguration impacting Tigera components, and denies unexpected traffic in case of defect or compromise. ### Ownership and management of allow-tigera[​](#ownership-and-management-of-allow-tigera) Tigera defines the `allow-tigera` tier and manages the policies within it. The Tigera Operator installs and monitors these policies, ensuring they always match the state defined by Tigera. Management by the Operator also ensures integrity for upgrades. > **SECONDARY:** The `allow-tigera` tier and its policies should not be edited, and the tier should not be moved. However, if you inadvertently make changes they are automatically reverted by the Operator to ensure your cluster is always protected. ## Tutorial[​](#tutorial) ### Change behavior of allow-tigera[​](#change-behavior-of-allow-tigera) If you want to change the way traffic is enforced by the `allow-tigera` tier, you must create policy in an adjacent tier to meet your needs. For example, if a policy in the `allow-tigera` tier allows or denies traffic, and you want to change how that traffic is enforced, you can create a policy in a tier before `allow-tigera` that selects the same traffic to make your desired changes. Similarly, if a policy in the `allow-tigera` tier passes or does not select traffic that you want to enforce, you can create a policy in a tier after `allow-tigera` to select this traffic to meet the desired behavior. ### Example: use preceding tier to tighten security[​](#example-use-preceding-tier-to-tighten-security) Let's say an `allow-tigera` policy allows ingress traffic from a Calico Cloud component that you do not use, and you want to tighten enforcement to not allow this traffic. Within a tier that comes before `allow-tigera`, you can create a policy that selects the same endpoint and contains ingress rules that deny traffic from that component and pass to `allow-tigera` for traffic from other components. ```yaml # allow-tigera.es-gateway-access allows ingress from deep packet inspection, a feature not utilized for the purpose of this example. # This policy tightens the scope of allowed ingress to es-gateway without modifying the allow-tigera policy directly. apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: preceding-tier.es-gateway-access namespace: tigera-elasticsearch spec: # Place in a tier prior to allow-tigera. tier: preceding-tier # Select the same endpoint as the original policy. selector: k8s-app == 'tigera-secure-es-gateway' ingress: # Select the same component ingress. - source: selector: k8s-app == 'tigera-dpi' namespaceSelector: name == 'tigera-dpi' # Enact different behavior (originally: Allow) action: Deny # Defer to allow-tigera for other ingress/egress decisions for this endpoint. - action: Pass ``` This example shows how you can change the impact of the `allow-tigera` tier on traffic without modifying the tier itself. This makes your changes more maintainable, and allows the allow-tigera tier to continue to receive updates as Calico Cloud evolves without you needing to reconcile your changes each release. For help to manage or change the behavior of the `allow-tigera` tier, contact Tigera Support. ### Configure RBAC for tiered policies ## Big picture[​](#big-picture) Configure fine-grained user access controls for tiered policies. ## Value[​](#value) Self-service is an important part of CI/CD processes for containerization and microservices. Calico Cloud provides fine-grained access control (RBAC) for: - Calico Cloud policy and tiers - Kubernetes network policy ## Concepts[​](#concepts) ### Standard Kubernetes RBAC[​](#standard-kubernetes-rbac) Calico Cloud implements the standard **Kubernetes RBAC Authorization APIs** with `Role` and `ClusterRole` types. The Calico Cloud API server integrates with Kubernetes RBAC Authorization APIs as an extension API server. ### RBAC for policies and tiers[​](#rbac-for-policies-and-tiers) In Calico Cloud, global network policy and network policy resources are associated with a specific tier. Admins can configure access control for these Calico Cloud policies using standard Kubernetes `Role` and `ClusterRole` resource types. This makes it easy to manage RBAC for both Kubernetes network policies and Calico Cloud tiered network policies. RBAC permissions include managing resources using the Calico Cloud web console, and `kubectl`. ### Fine-grained RBAC for policies and tiers[​](#fine-grained-rbac-for-policies-and-tiers) RBAC permissions can be split by resources (Calico Cloud and Kubernetes), and by actions (CRUD). Tiers should be created by administrators. Full CRUD operations on tiers is synonymous with full management of network policy. Full management to network policy and global network policy also requires `GET` permissions to 1) any tier a user can view/manage, and 2) the required access to the tiered policy resources. Here are a few examples of how you can fine-tune RBAC for tiers and policies. | **User** | **Permissions** | | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Admin | The default **tigera-network-admin** role lets you create, update, delete, get, watch, and list all Calico Cloud resources (full control). Examples of limiting Admin access:* List tiers only * List only specific tiers | | Non-Admin | The default **tigera-ui-user** role allows users to only list Calico Cloud policy and tier resources. Examples of limiting user access:* Read-only access to all policy resources across all tiers, but only write access for NetworkPolicies with a specific tier and namespace. * Perform any operations on NetworkPolicies and GlobalNetworkPolicies. * List tiers only. * List or modify any policies in any tier. Fully manage only Kubernetes network policies in the **default** tier, in the **default** namespace, with read-only access for all other tiers. | ### RBAC definitions for Calico Cloud network policy[​](#rbac-definitions-for-calico-cloud-network-policy) To specify per-tier RBAC for the Calico Cloud network policy and Calico Cloud global network policy, use pseudo resource kinds and names in the `Role` and `ClusterRole` definitions. For example, ```yaml kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: tier-default-reader rules: - apiGroups: ['projectcalico.org'] resources: ['tiers'] resourceNames: ['default'] verbs: ['get'] - apiGroups: ['projectcalico.org'] resources: ['tier.networkpolicies', 'networkpolicies] resourceNames: ['default.*'] verbs: ['get', 'list'] ``` Where: - **resources**: both `tier.globalnetworkpolicies` and `globalnetworkpolicies` for global network policies, and both `tier.networkpolicies` and `networkpolicies` for network policies. - **resourceNames**: - Blank - any policy of the specified kind across all tiers. - `.*` - any policy of the specified kind within the named tier. - `` - the specific policy of the specified kind. Because the policy name is prefixed with the tier name, this also specifies the tier. ## Before you begin...[​](#before-you-begin) **Required** A **cluster-admin** role with full permissions to create and modify resources. **Recommended** A rough idea of your tiered policy workflow, and who should access what. See [Configure tiered policies](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/tiered-policy). ## How to[​](#how-to) - [Create Admin users, full permissions](#create-admin-users-full-permissions) - [Create minimum permissions for all non-Admin users](#create-minimum-permissions-for-all-non-admin-users) > **SECONDARY:** `kubectl auth can-i` cannot be used to check RBAC for tiered policy. ### Create Admin users, full permissions[​](#create-admin-users-full-permissions) Create an Admin user with full access to the Calico Cloud web console (as well as everything else in the cluster) using the following command. See the Kubernetes documentation to identify users based on your chosen [authentication method](https://kubernetes.io/docs/reference/access-authn-authz/authentication/), and how to use the [RBAC resources](https://kubernetes.io/docs/reference/access-authn-authz/rbac/). ```text kubectl create clusterrolebinding permissive-binding \ --clusterrole=cluster-admin \ --user= ``` ### Create minimum permissions for all non-Admin users[​](#create-minimum-permissions-for-all-non-admin-users) All users using the Calico Cloud web console should be able to create authorizationreviews and authorizationrequests as well as access license information through the services/proxy https:calico-api:8080. 1. Download the [min-ui-user-rbac.yaml manifest](https://docs.tigera.io/files/min-ui-user-rbac.yaml). The roles and bindings in this file provide a minimum starting point for setting up RBAC for your users according to your specific security requirements. This manifest provides basic RBAC to view some statistical data in the UI but does not provide permissions to view or modify any network policy related configuration. 2. Run the following command to replace `` with the name or email of the user you are providing permissions to: ```bash sed -i -e 's///g' min-ui-user-rbac.yaml ``` 3. Use the following command to install the bindings: ```bash kubectl apply -f min-ui-user-rbac.yaml ``` ## Tutorial[​](#tutorial) This tutorial shows how to use RBAC to control access to resources and CRUD actions for a non-Admin user, John, with the username **john**. The RBAC examples shown will include: - [User cannot read policies in any tier](#user-cannot-read-policies-in-any-tier) - [User can view all policies, and modify policies in the default namespace and tier](#user-can-view-all-policies-and-modify-policies-in-the-default-namespace-and-tier) - [User can read policies only in both the default tier and namespace](#user-can-read-policies-only-in-both-the-default-tier-and-namespace) - [User can read policies only in both a specific tier and in the default namespace](#user-can-read-policies-only-in-both-a-specific-tier-and-in-the-default-namespace) - [User can only view a specific tier](#user-can-only-view-a-specific-tier) - [User can read all policies across all tiers and namespaces](#user-can-read-all-policies-across-all-tiers-and-namespaces) - [User has full control over policies only in both a specific tier and in the default namespace](#user-has-full-control-over-policies-only-in-both-a-specific-tier-and-in-the-default-namespace) ### User cannot read policies in any tier[​](#user-cannot-read-policies-in-any-tier) User 'john' is forbidden from reading policies in any tier (**default** tier, and **net-sec** tier). When John issues the following command: ```bash kubectl get networkpolicies.p ``` It returns: ```text Error from server (Forbidden): networkpolicies.projectcalico.org is forbidden: User "john" cannot list networkpolicies.projectcalico.org in tier "default" and namespace "default" (user cannot get tier) ``` Similarly, when John issues this command: ```bash kubectl get networkpolicies.p -l projectcalico.org/tier==net-sec ``` It returns: ```text Error from server (Forbidden): networkpolicies.projectcalico.org is forbidden: User "john" cannot list networkpolicies.projectcalico.org in tier "net-sec" and namespace "default" (user cannot get tier) ``` > **SECONDARY:** The .p' extension (`networkpolicies.p`) is short for "networkpolicies.projectcalico.org" and used to differentiate it from the Kubernetes NetworkPolicy resource and the underlying CRDs (if using the Kubernetes Datastore Driver). > **SECONDARY:** The label for selecting a tier is `projectcalico.org/tier`. When a label selector is not specified, the server defaults the selection to the `default` tier. Alternatively, a field selector (`spec.tier`) may be used to select a tier. > > ```bash > kubectl get networkpolicies.p --field-selector spec.tier=net-sec > ``` ### User can view all policies, and modify policies in the default namespace and tier[​](#user-can-view-all-policies-and-modify-policies-in-the-default-namespace-and-tier) 1. Download the [`read-all-crud-default-rbac.yaml` manifest](https://docs.tigera.io/files/read-all-crud-default-rbac.yaml). 2. Run the following command to replace `` with the `name or email` of the user you are providing permissions to: ```bash sed -i -e 's///g' read-all-crud-default-rbac.yaml ``` 3. Use the following command to install the bindings: ```bash kubectl apply -f read-all-crud-default-rbac.yaml ``` The roles and bindings in this file provide the permissions to read all policies across all tiers and to fully manage policies in the **default** tier and **default** namespace. This file includes the minimum required `ClusterRole` and `ClusterRoleBinding` definitions for all UI users (see `min-ui-user-rbac.yaml` above). ### User can read policies only in both the default tier and namespace[​](#user-can-read-policies-only-in-both-the-default-tier-and-namespace) In this example, we give user 'john' permission to read policies only in both the **default** tier and namespace. ```yaml kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: tigera-example-get-default-tier rules: # To access Calico policy in a tier, the user requires "get" access to that tier. - apiGroups: ["projectcalico.org"] resources: ["tiers"] resourceNames: ["default"] verbs: ["get"] --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: tigera-example-read-policies-in-default-tier rules: # This allows "get" and "list" of the Calico NetworkPolicy resources in the default tier. - apiGroups: ["projectcalico.org"] resources: ["tier.networkpolicies"] resourceNames: ["default.*"] verbs: ["get", "list"] --- # tigera-example-get-default-tier is applied globally kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: john-can-get-default-tier subjects: - kind: User name: john apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: tigera-example-get-default-tier apiGroup: rbac.authorization.k8s.io --- # tigera-example-read-policies-in-default-tier is applied per-namespace kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: john-can-read-policies-in-default-tier-and-namespace subjects: - kind: User name: john apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: tigera-example-read-policies-in-default-tier apiGroup: rbac.authorization.k8s.io ``` With the above, user john is able to list all NetworkPolicy resources in the **default** tier: ```bash kubectl get networkpolicies.p --all-namespaces ``` With some example policies on the cluster, returns: ```text NAMESPACE NAME CREATED AT blue default.calico-np-blue-ns-default-tier 2021-07-26T09:05:11Z default default.calico-np-default-ns-default-tier 2021-07-26T09:05:11Z green default.calico-np-green-ns-default-tier 2021-07-26T09:05:13Z red default.calico-np-red-ns-default-tier 2021-07-26T09:05:12Z yellow default.calico-np-yellow-ns-default-tier 2021-07-26T09:05:13Z ``` As intended, user john can only examine those in the **default** namespace: ```bash kubectl get networkpolicies.p default.calico-np-green-ns-default-tier -o yaml -n=green ``` Correctly returns: ```text Error from server (Forbidden): networkpolicies.projectcalico.org "default.calico-np-green-ns-default-tier" is forbidden: User "john" cannot get networkpolicies.projectcalico.org in tier "default" and namespace "green" ``` John also still cannot access tier **net-sec**, as intended: ```bash kubectl get networkpolicies.p -l projectcalico.org/tier==net-sec ``` This returns: ```text Error from server (Forbidden): networkpolicies.projectcalico.org is forbidden: User "john" cannot list networkpolicies.projectcalico.org in tier "net-sec" and namespace "default" (user cannot get tier) ``` ### User can read policies only in both a specific tier and in the default namespace[​](#user-can-read-policies-only-in-both-a-specific-tier-and-in-the-default-namespace) Let's assume that the kubernetes-admin gives user 'john' the permission to list the policies in tier **net-sec**, but only examine the detail of the policies that are also in the **default** namespace. To provide these permissions to user 'john', use the following `ClusterRoles`,`ClusterRoleBinding` and `RoleBinding`. ```yaml kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: tigera-example-get-net-sec-tier rules: # To access Calico policy in a tier, the user requires "get" access to that tier. - apiGroups: ["projectcalico.org"] resources: ["tiers"] resourceNames: ["net-sec"] verbs: ["get"] --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: tigera-example-read-policies-in-net-sec-tier rules: # This allows "get" and "list" of the Calico NetworkPolicy resources in the net-sec tier. - apiGroups: ["projectcalico.org"] resources: ["tier.networkpolicies"] resourceNames: ["net-sec.*"] verbs: ["get", "list"] --- # tigera-example-get-net-sec-tier is applied globally kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: john-can-get-net-sec-tier subjects: - kind: User name: john apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: tigera-example-get-net-sec-tier apiGroup: rbac.authorization.k8s.io --- # tigera-example-read-policies-in-net-sec-tier is applied per-namespace kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: john-can-read-policies-in-net-sec-tier-and-namespace subjects: - kind: User name: john apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: tigera-example-read-policies-in-net-sec-tier apiGroup: rbac.authorization.k8s.io ``` ### User can only view a specific tier[​](#user-can-only-view-a-specific-tier) In this example, the following `ClusterRole` and `ClusterRoleBinding` can be used to provide 'get' access to the **net-sec** tier. This has the effect of making the **net-sec** tier visible in the Calico Cloud web console (including listing the names of the policies it contains). However, to modify or view the details of policies within the **net-sec** tier, additional RBAC permissions would be required. ```yaml kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: tigera-example-make-net-sec-tier-visible rules: # To access Calico policy in a tier, the user requires "get" access to that tier. - apiGroups: ["projectcalico.org"] resources: ["tiers"] resourceNames: ["net-sec"] verbs: ["get"] --- # tigera-example-make-net-sec-tier-visible is applied globally kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: john-can-view-the-net-sec-tier subjects: - kind: User name: john apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: tigera-example-make-net-sec-tier-visible apiGroup: rbac.authorization.k8s.io ``` ### User can read all policies across all tiers and namespaces[​](#user-can-read-all-policies-across-all-tiers-and-namespaces) In this example, a single `ClusterRole` is used to provide read access to all policy resource types across all tiers. In this case, there is no need to use both `ClusterRoleBindings` and `RoleBindings` to map these abilities to the target user, because the intention is to for the policy to apply to all current and future namespaces on the cluster, so a `ClusterRoleBinding` provides the desired granularity. ```yaml kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: tigera-example-all-tiers-and-namespaces-policy-reader rules: # To access Calico policy in a tier, the user requires "get" access to that tier. # Not specifying any specific "resourceNames" provides access to all tiers. - apiGroups: ["projectcalico.org"] resources: ["tiers"] verbs: ["get"] # This allows read access to the Kubernetes NetworkPolicy resources (these are always in the default tier). - apiGroups: ["networking.k8s.io", "extensions"] resources: ["networkpolicies"] verbs: ["get","watch","list"] # This allows read access to the Calico NetworkPolicy and GlobalNetworkPolicies. # Not specifying any specific "resourceNames" provides access to them in all tiers. - apiGroups: ["projectcalico.org"] resources: ["tier.networkpolicies","tier.globalnetworkpolicies"] verbs: ["get","watch","list"] --- # tigera-example-all-tiers-and-namespaces-policy-reader is applied globally, with a single ClusterRoleBinding, # since all the rules it contains apply to all current and future namespaces on the cluster. kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: read-all-tier subjects: - kind: User name: john apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: tigera-example-all-tiers-and-namespaces-policy-reader apiGroup: rbac.authorization.k8s.io ``` ### User has full control over policies only in both a specific tier and in the default namespace[​](#user-has-full-control-over-policies-only-in-both-a-specific-tier-and-in-the-default-namespace) In this example, two `ClusterRole` objects are used to provide full access control of Calico NetworkPolicy resource types in the **net-sec** tier: - The `tiers` resource is bound to a user using a `ClusterRoleBinding`, because it is a global resource. This results in the user having the ability to read the contents of the tier across all namespaces. - The `networkpolicies` resources are bound to a user using a `RoleBinding`, because the aim in this case was to make them CRUD-able only in the default namespace. You only need this one `ClusterRole` to be defined, but it can be applied to different namespaces using additional `RoleBinding` objects. If the intention was to apply it to all current and future namespaces, a `ClusterRoleBinding` could be used. ```yaml kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: tigera-example-get-net-sec-tier rules: # To access Calico policy in a tier, the user requires "get" access to that tier. - apiGroups: ["projectcalico.org"] resources: ["tiers"] resourceNames: ["net-sec"] verbs: ["get"] --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: tigera-example-crud-policies-in-net-sec-tier rules: # This allows full CRUD access to the Calico NetworkPolicy resources in the net-sec tier. - apiGroups: ["projectcalico.org"] resources: ["tier.networkpolicies"] resourceNames: ["net-sec.*"] verbs: ["*"] --- # tigera-example-get-net-sec-tier is applied globally kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: john-can-get-net-sec-tier subjects: - kind: User name: john apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: tigera-example-get-net-sec-tier apiGroup: rbac.authorization.k8s.io --- # tigera-example-crud-policies-in-net-sec-tier is applied per-namespace kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: john-can-crud-policies-in-net-sec-tier-and-namespace subjects: - kind: User name: john apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: tigera-example-crud-policies-in-net-sec-tier apiGroup: rbac.authorization.k8s.io ``` ### Get started with network sets ## Visualize traffic to/from your cluster[​](#visualize-traffic-tofrom-your-cluster) Modern applications often integrate with third-party APIs and SaaS services that live outside Kubernetes clusters. To securely enable access to those integrations, you must be able to limit IP ranges for egress and ingress traffic to workloads. Limiting IP lists or ranges is also used to deny-list bad actors or embargoed countries. To limit IP ranges, you need to use the Calico Cloud resource called **network sets**. ## What are network sets?[​](#what-are-network-sets) **Network sets** are a grouping mechanism that allows you to create an arbitrary set of IP subnetworks/CIDRs or domains that can be matched by standard label selectors in Kubernetes or Calico Cloud network policy. Like IP pools for pods, they allow you to reuse/scale sets of IP addresses in policies. A **network set** is a namespaced resource that you can use with Kubernetes or Calico Cloud network policies; a **global network set** is a cluster-wide resource that you can use with Calico Cloud network policies. Like network policy, you manage user access to network sets using standard Kubernetes RBAC. ## Why are network sets powerful?[​](#why-are-network-sets-powerful) If you are familiar with Service Graph in the web console, you know the value of seeing pod-to-pod traffic within your cluster. But what about traffic external to your cluster? Calico Cloud automatically detects IPs for pods and nodes that fall into the standard IETF “public network” and “private network” designations, and displays those as icons in Service Graph. So you get some visibility into external traffic without using any network sets. ![public-private-networks](https://docs.tigera.io/assets/images/public-private-networks-58c5e15f90f09422c523d29fff1fc73b.png) However, when you create network sets, you can get more granular visibility into what's leaving the cluster to public networks. Because you control the grouping, the naming, and labeling, you create visibility that is customized to your organization. This is why they are so powerful. Here are just a few examples of how network sets can be used: - **Egress access control** Network sets are a key resource for defining egress access controls; for example, securing ingress to microservices/apps or egress from workloads outside the cluster. - **Troubleshooting** Network sets appear as additional metadata in flow logs and Kibana, Flow Visualizer, and Service Graph. - **Efficiency and scaling** Network sets are critical when scaling your deployment. You may have only a few CIDRs when you start. But as you scale out, it is easier to update a handful of network sets than update each network policy individually. Also, in a Kubernetes deployment, putting lots of anything (CIDRs, ports, policy rules) directly into policies causes inefficiencies in traffic processing (iptables/eBPF). - **Microsegmentation and shift left** Network sets provide the same microsegmentation controls as network policy. For example, you can allow specific users to create policies (that reference network sets), but allow only certain users to manage network sets. - **Threat defense** Network sets are key to being able to manage threats by blocking bad IPs with policy in a timely way. Imagine having to update individual policies when you find a bad IP you need to quickly block. You can even give access to a controller that automatically updates CIDRs in a network set when a bad IP is found. ## Create a network set and use it in policy[​](#create-a-network-set-and-use-it-in-policy) In this section, we’ll walk through how to create a namespaced network set in the web console. You can follow along using your cluster or tigera-labs cluster. In this example, you will create a network set named, `google`. This network set contains a list of trusted google endpoints for a microservice called, `hipstershop`. As a service owner, you want to be able to see traffic leaving the microservices in Service Graph. Instead of matching endpoints on IP addresses, we will use domain names. 1. From the left navbar, click **Network Sets**. 2. Click **Add Network Set**, and enter these values. - For Name: `google` - For Scope: Select **Namespace** and select, `hipstershop` 3. Under Labels, click **Add label**. - In the Select key field, enter `destinations` and click the green bar to add this new entry. - In the Value field, enter `google`, click the green bar to add the entry, and save. 4. For Domains, click **+Add Domain** and these URLs: `clouddebugger.googleapis.com`, `cloudtrace.googleapis.com`, `metadata.google.internal`, `monitoring.googleapis.com`. 5. Click **Create Network Set**. You’ve created your first network set. ![add-networkset-google](https://docs.tigera.io/assets/images/add-networkset-google-383ea2b92cdea32c79367d937e2e5dca.png) The YAML looks like this: ```yaml kind: NetworkSet apiVersion: projectcalico.org/v3 metadata: name: google labels: destinations: google namespace: hipstershop spec: nets: [] allowedEgressDomains: - clouddebugger.googleapis.com - cloudtrace.googleapis.com - metadata.google.internal - monitoring.googleapis.com ``` Next, we write a DNS policy for hipstershop that allows egress traffic to the trusted google sites. The following network policy allows egress access for all destination selectors labeled, `google`. Note that putting domains in a network set referencing it in policy is the best practice. Also, note that using `selector: all()` should only be used if all pods in the namespace can access all of the domains in the network set; if not, you should create separate policies accordingly. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: application.allow-egress-domain namespace: hipstershop spec: tier: application order: 0 selector: all() serviceAccountSelector: '' egress: - action: Allow source: {} destination: selector: destinations == "google" types: - Egress ``` ## Network sets in Service Graph[​](#network-sets-in-service-graph) Continuing with our `hipstershop` example, if you go to Service Graph, you see hipstershop (highlighted in yellow). ![hipstershop](https://docs.tigera.io/assets/images/hipstershop-869531dc562c3a6c74653462edd412b4.png) If we double-click `hipstershop` to drill down, we now see the `google` network set icon (highlighted in yellow). We now have visibility to traffic external from google sites to hipstershop. (If you are using the tigera-labs cluster, note that the network set will not be displayed as shown below.) ![google-networkset](https://docs.tigera.io/assets/images/google-networkset-1d5879d8426568a6dae6e11f905ef609.png) Service Graph provides a view into how services are interconnected in a consumable view, along with easy access to flow logs. However, you can also see traffic associated with network sets in volumetric display with Flow Visualizer, and query flow log data associated with network sets in Kibana. ## Tutorial[​](#tutorial) In the following example, we create a global network set resource for a trusted load-balancer that can be used with microservices and applications. The label, `trusted-ep: load-balancer` is how this global network set can be referenced in policy. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkSet metadata: name: load-balancer labels: trusted-ep: "load-balancer" spec: nets: # Modify the ip addresses to refer to the ip addresses of load-balancers in your environment - 10.0.0.1/32 - 10.0.0.2/32 ``` The following network policy uses the `selector: trusted-ep == "load balancer"` to reference the above GlobalNetworkSet. All applications in the `app2-ns` namespace, that match `app2` and `svc1` are allowed ingress traffic from the trusted load balance on port 1001. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: application.app2-svc1 namespace: app2-ns spec: tier: application order: 500 selector: (app == "app2"&&svc == "svc1") ingress: - action: Allow protocol: TCP source: selector: trusted-ep == "load-balancer" destination: ports: - '10001' types: - Ingress ``` ### Advanced policy rules with network sets[​](#advanced-policy-rules-with-network-sets) When you combine Calico Cloud policy rules with network sets, you have powerful ways to fine-tune. The following example combines network sets with specific rules in a global network policy to deny access more quickly. We start by creating a Calico Cloud GlobalNetworkSet that specifies a list of CIDR ranges we want to deny: 192.0.2.55/32 and 203.0.113.0/24. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkSet metadata: name: ip-protect labels: ip-deny-list: 'true' spec: nets: - 192.0.2.55/32 - 203.0.113.0/24 ``` Next, we create two Calico Cloud GlobalNetworkPolicy resources. The first is a high "order" policy that allows traffic as a default for things that don’t match our second policy, which is low "order" and uses the GlobalNetworkSet label as a selector to deny ingress traffic (IP-deny-list in the previous step). In the label selector, we also include the term, `!has(projectcalico.org/namespace)`, which prevents this policy from matching pods or NetworkSets that also have this label. To more quickly enforce the denial of forwarded traffic to the host at the packet level, use the `doNotTrack` and `applyOnForward` options. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: forward-default-allow spec: selector: apply-ip-protect == 'true' order: 1000 doNotTrack: true applyOnForward: true types: - Ingress ingress: - action: Allow --- apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: ip-protect spec: selector: apply-ip-protect == 'true' order: 0 doNotTrack: true applyOnForward: true types: - Ingress ingress: - action: Deny source: selector: ip-deny-list == 'true' && !has(projectcalico.org/namespace) ``` ## Best practices for using network sets[​](#best-practices-for-using-network-sets) - Create network sets as soon as possible after getting started This allows you to quickly realize the benefits of seeing custom metadata in flow logs and visualizing traffic in Service Graph and Flow Visualizer. - Create a network set label and name schema It is helpful to think: what names would be meaningful and easy to understand when you look in Service Graph? Flow Viz? Kibana? What labels will be easy to understand when used in network policies – especially if you are separating users who manage network sets from those who consume them in network policies. - Do not put large sets of CIDRs and domains directly in policy Network sets allow you to specify CIDRs and/or domains. Although you can add CIDRs and domains directly in policy, it doesn't scale. - Do not put thousands of rules into a policy, each with a different CIDR If your set of /32s can be easily aggregated into a few broader CIDRs without compromising security, it’s a good thing to do; whether you’re putting the CIDRs in the rule or using a network set. - If you want to match thousands of endpoints, write one or two rules and use selectors to match the endpoints. Having one rule per port, per host is inefficient because each rule ends up being rendered as an iptables/eBPF rule instead of making good use of IP sets. - Avoid overlapping IP addresses/subnets in networkset/globalnetworkset definitions ## Efficient use of network sets[​](#efficient-use-of-network-sets) If you have a large number of things to match, using a network set is more efficient both in the control plane (for example, Felix CPU), and for the packet path (latency/per packet CPU). If you use network sets and you add/remove an IP from the network set, this doesn't require changing iptables rules at all. It only requires updating the ipset, which is efficient. If you also change the policy rules, then iptables must be updated too. Using network sets is efficient for all of the following use cases: - The system applying the iptables rules to incoming connections (to decide whether to allow or deny the traffic) - iptables rules updates whenever one of the policies and/or network sets change - The Kubernetes APIserver handling changes to the policy and/or networkset CRDs Follow these guidelines for efficient use of network sets. | Policy | Network set | Results | | ----------------------------------------------------------------------------------------------------- | -------------------------------------- | ----------------------------------------------------------------------------------------------------- | | source: selector: foo="bar" | With handful of broad CIDRs | **Efficient** **-** 1 iptables/eBPF rule - 1 IP set with handful of CIDRs | | source: nets: \[ ... handful ...] | Not used | **Efficient** - Handful of iptables/eBPF rules - 0 IP sets | | source: selector: foo="bar" | One network set with 2000 x /32s | **`*`Most efficient** - 1 iptables/eBPF rule - 1 IP sets with 2000 entries | | | Two network sets with 1000 each x /32s | **Efficient** - 2 iptables/eBPF rules - 2 IP set with 1000 entries | | source: nets: \[... 2000 /32s ...] - source: nets: \[1 x /32] - source: nets: \[1 x /32] - ... x 2000 | Not used | **Inefficient** Results in programming 2k iptables/eBPF rules - 2000+ iptables/eBPF rules - 0 IP sets | `*` Updating **ipsets** is fast and efficient. Adding/removing a single entry is an O(1) operation no matter the number of IPs in the set. Updating **iptables** is generally slow and gets slower the more rules you have in total (including rules created by kube-proxy, for example). Less than 10K rules are generally fine, but noticeable latency occurs when updating rules above that number (increasing as the number of rules grows). (The newer nftables may scale more efficiently but those results are not included here.) Similarly, hitting many iptables (or eBPF) rules adds latency to the first packet in a flow. For iptables, it measures around 250ns per rule. Although a single rule is negligible, hitting 10K rules add >1ms to the first packet in the flow. Packets only hit the rules for the particular interface that they arrive on; if you have 10K rules on one interface and 10 rules on another, the packet processed by the first interface will have more latency. ## Additional resources[​](#additional-resources) - [Network set](https://docs.tigera.io/calico-cloud/reference/resources/networkset) - [Global network set](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkset) ### Global default deny policy best practices In this article you will learn when and how to create a global default deny policy for the cluster. ### What is it, when should you create one?[​](#what-is-it-when-should-you-create-one) A global default deny policy ensures that unwanted traffic (ingress and egress) is denied by default. Pods without policy (or incorrect policy) are not allowed traffic until appropriate network policy is defined. Although the staging policy tool will help you find incorrect and missing policy, a global deny helps mitigate against other lateral malicious attacks. ### Best practice #1: Allow, stage, then deny[​](#best-practice-1-allow-stage-then-deny) We recommend that you create a global default deny policy *after you complete writing policy for the traffic that you want to allow*. Use the stage policy feature to get your allowed traffic working as expected, then lock down the cluster to block unwanted traffic. The following steps summarizes the best practice: 1. Create a staged global default deny policy. It will show all the traffic that would be blocked if it were converted into a deny. 2. Create other network policies to individually allow the traffic shown as blocked in step 1, until no connections are denied. 3. Convert the staged global network policy to an enforced policy. ### Best practice #2: Keep the scope to non-system pods[​](#best-practice-2-keep-the-scope-to-non-system-pods) A global default deny policy applies to the entire cluster including all workloads in all namespaces, hosts (computers that run the hypervisor for VMs, or container runtime for containers), including Kubernetes control plane and Calico Cloud control plane nodes and pods. For this reason, the best practice is to create a global default deny policy for **non-system pods** as shown in the following example. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: deny-app-policy spec: namespaceSelector: has(projectcalico.org/name) && projectcalico.org/name not in {"kube-system", "calico-system"} types: - Ingress - Egress egress: # allow all namespaces to communicate to DNS pods - action: Allow protocol: UDP destination: selector: 'k8s-app == "kube-dns"' ports: - 53 - action: Allow protocol: TCP destination: selector: 'k8s-app == "kube-dns"' ports: - 53 ``` Note the following: - Even though we call this policy "global default deny", the above policy is not explicitly *denying traffic*. By selecting the traffic with the `namespaceSelector` but not specifying an allow, the traffic is denied after all other policy is evaluated. This design also makes it unnecessary to ensure any specific order (priority) for the default-deny policy. - Allowing access to `kube-dns` simplifies per-pod policies because you don't need to duplicate the DNS rules in every policy - The policy deliberately excludes the `kube-system`, and `calico-system` namespaces by using a negative `namespaceSelector` to avoid impacting any control plane components Next, add the policy to the default tier. (As noted above, anywhere in the default tier is fine.) Next, use the stage policy feature and verify that the policy does not block any necessary traffic before enforcing it. ### Don't try this\![​](#dont-try-this) The following policy looks fine on the surface, and it does work. But as described in Best practices #2, the policy could break your cluster because the scope is too broad. Therefore, we do not recommend adding this type of policy to the default tier, even if you have verified allowed traffic using the stage policy feature. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: default.default-deny spec: tier: default selector: all() types: - Ingress - Egress ``` ### Stage, preview impacts, and enforce policy ## Big picture[​](#big-picture) Stage and preview impacts on traffic before enforcing policy. ## Value[​](#value) Calico Cloud staged network policy resources lets you test the traffic impact of the policy as if it were enforced, but without changing traffic flow. You can also preview the impacts of a staged policy on existing traffic. By verifying that correct flows are allowed and denied before enforcement, you can minimize misconfiguration and potential network disruption. ## Concepts[​](#concepts) ### About staged policies[​](#about-staged-policies) The following staged policy resources have the same structure (i.e. the resource spec has the same fields) as their “enforced” counterpart. - Staged global network policy - Staged network policy - Staged Kubernetes network policy ### Review permissions[​](#review-permissions) The default `tigera-network-admin` cluster role has the required permissions to manage the different enforced and staged network policies. Adjust permissions for your environment. As with Calico Cloud network policy and global network policies, the RBAC for Calico Cloud staged network policy and staged global network policy is tier-dependent. ## How to[​](#how-to) - [Create a policy recommendation](#create-a-policy-recommendation) - [Stage a policy](#stage-a-policy) - [Preview policy impact](#preview-policy-impact) - [Enforce a staged policy](#enforce-a-staged-policy) - [Stage updates to an enforced policy](#stage-updates-to-an-enforced-policy) ### Create a policy recommendation[​](#create-a-policy-recommendation) One of the first things developers need to do is secure unprotected workloads with network policy. (For example, by default, Kubernetes pods accept traffic from any source.) The **Recommend policy** feature allows developers with minimal experience writing policy to secure workloads. Because **Recommend policy** looks at historical flow log entries that match your request, you should run your workloads for a reasonable amount of time to get "typical network traffic" for your application. 1. In the left navbar, click **Policies**. 2. Click **Recommend a policy**. 3. Enter time range, Namespace, Name, and click **Recommend**. 4. If relevant flow logs are found within the time range for the workload endpoint, click **Preview** to assess the impact of the recommended policy, or **Stage**. ![recommend-policy](https://docs.tigera.io/assets/images/recommend-policy-61392143274f8323a197cc58822d1515.png) ### Stage a policy[​](#stage-a-policy) Stage a policy to test it in a near replica of a production environment. 1. In the left navbar, click **Policies**. 2. In a tier, click **Add Policy**. 3. Create your policy and click **Stage** to save and stage it. ![stage-new-policy](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAVEAAABNCAYAAAAfFVcWAAAAAXNSR0IArs4c6QAAFqJJREFUeAHtnQd8VUX2x0+Sl4SEkEIKBEJIIAQIvVelSBUVBaTq3/ZXPxbcdV1dlXUtrCvih3VdFdl1WUXWP2sDRLqCSC+h95ICoQRSgPSElP/5Tbzwcnl5efe9l/Y48/kkeffeuXNnvnfmzDlnzry45eXllpEkISAEhIAQsIuAqaio2K4b5SYhIASEgBAgMmVeviochIAQEAJCwE4CbtnZOWLO2wlPbhMCQkAImNzd3YSCEBACQkAI2EnA3c775DYhIASEgBBgAiJEpRsIASEgBBwgIELUAXhyqxAQAkLAJAiEgBCwTMAr80HLF26xs0WNF95iLTbWXNFEjfGS3EJACAiBCgREiFbAIQdCQAgIAWMERIga4yW5hYAQEAIVCDjsE3VTYaauGmtaRmU2bkVwKwdRAa7hA36YjY8zXLQjN7j2OyZ+x7ZRvxYM36Dr9/WqfMFO6euOdMhqvdf2Ma9VwyEhajK5U3Z2IeUVFGnludRfXx8vauTnTcXFpZW2CwLGw92dLl/NJ/U9BHaOMTcenAH+Dcjby0TFJZU/T6uIrQNfy2/pry2DweThTvmF1ygru8BSEfX7HMtOLy8PCgrwpZLS0ionzPz8a5Sb75p93aeBJ/k3amC1r+Nle3l6UH7BNSq6VlK/330ltceY92S5dq3Y9vbZLUSx02nrziRKv5xPfg29XW5+hm6Sk1NIoSG+1LdHSyopsaytlJSW0bpNxxl6Gfl4e5I9CilKLuNyrmTnU9cOzah1y2CrgrSUBzySLUJQZbTwC0IYP+48AVSWIECPnbpER05cZAHvq9pm5xxR2SNq9TwU0AKeIIBgyIAYglJgTSldt+kUNWJB40oM8ALQ/3LziigwwJsG9Iqu9J14cH9YvPIAffltPF2+kmdXX6+08Fq+oI3uVpEh9Mxjt/E4bG6zILVLiALmiYRLlJ1XTOPv6ubQYK5ldlYfD8H2w9qDdCopnWKiQ1mQVtQQPU0etGVXEnc+PxrYp7XVsmy5mJ1TQMvWHKCwED/CjGhJ24QA9fDwcApzlF9SUmJRkGKSzOCBcuhYKt03phtBU3HVtGN3Mu3ce4Zu69vKqiY27u7ubHW4mgi98VZX/nRITZodA2+c0z55sga6hZWmP81aSddYC/XwcE0Op1Mu09kLV+iLjx6gxkG+VMoyoKpUuRpi5U4APH32Mmto0U4ZzFYeVauX3HjA9OE2oq2YOPQJAu1SRo7ioL9mz3EjvwbUOiqUzp6/YrGTakLVEQ3UvF5aOVq55tfQ3jPc7ri2zVxagKLNvbq1pAy2qPSTpDkPfHZlAYr29e3ZilLOXcbHmxK09I3bTilTHi4Q9A9X/PH18aSkMxnK+oKSZEu6WTLYchcbNBh4AOvqCW381Xq+qanlcxR7M52onXhy57Rl9rupMtVwAvWABuLqCZOJNqG4eluttQ/uG2uKV1FRCXOyVoJrXIMFCm3bVr+NQ1LQmv/INXBi5bZmW1LDj6uycTXd/iorJBmqjUBV7/pWEKAaXCNtdUiIag+Uv9YJYOX3/Xnr6NDR89YzVtPV4uJiZTmItlVNgKVYqwRgtWI1H9Er+KnM0irlfEZWxa0+tAYv1qoQTWbfw7b4RDqfWr3frr+CHeZ79p+pQawVH1XG61HreQX/eMLFihecdIQV9sp+TCYTZWVlUXJyco0JUqz0Hjl+nvYfSqHUS1l2txKLlzP+sowHoOv8C5vTKZm0c0/y9ZCxPGaVxn51V00QoD4NvKh/r1Y0bFB7Gja4PQU3bqiiTyBM4YdGHi3fnDcn0PTHh1AhC1uc066DDx8qAYz7cA0JCorel41zlQlqdZOTf9m1Ou9oHQDgn19soWWr91PTJv5qAWPCPd3p8QcGsDBwU+EWGDj+vNACmF4cO3mN/THFvJKMmM2gQN/rVcCAzc0r5BXtRhXOFXAsG14W0gpeYe/aMYK6d4m8nqemP6ANiCetjpSens4+nGsW/XrQPnNycgh58DkyMtLiarwz67X3QAq99NZi6sG8z/BCxbTxvWncXV2VnwkuNQ/2M8NpjzAZfMZ7zuP4S7z7BhwmhpSTW0je3ia10KZxQ6wqBgjiOtEvMHgQxYAwJSTcixhGd24n7q1raenK/TT/y60qHvMqh7N99sGDtGHrSVq74SjNeXO8Oo+xkZ6Zq/p4YWGxWtQDF/BA3/fjuGVoc2g3UsblXG63iRr6ete15qr64B1FNAuiubMnc6THBcrjsZp5JVe9P7ynAH8fNYlcuZpHY4Z3otHD4mj6K1+zECxlf7wXNQn1p0vp2XxfETXkUEq0G2Xi/eNc83AOJeBOdY5X1FEe+IU3CVB5LqZl8eIXIlmqF02t9DTEVX77wx6az50osnkQxbOWuGHzCaXy/7L1BH00/xcK4Hg8OHfj2jWjSfd0o9+9vphCgxtReka2Cqt6dFp/Wr7mIH3+1TYq4RjNNjFhNPPlu5XG98VX25XAaB/blF7+zUgeUJ68CFbzCyR4+QGNfFSAsvlrhHaGiQCB9Y4mdJrc3FweWEWqzTg2TxCchYWFqnPl5+erzmktNtT8Xns/w3SLbN6YPnxnEi1dtZ8+/+82atmiMb3+7nLV7pmv3E1r1h+hLTsSeCGjlO6/pyf5+XqpcLKPZk0iaGvQQJ/8n4G0Y3cSPTKlL33x9Q5axRYF0h23t6P2seH0n2920CfvTaG3/7paCeE5b42nl95cQtMm9HJaxIS9DCzdt5wn834ch/nqb0fxxH5IBe6v536/izXTeQs20oMT+9Jr7/zAk0uuEqhZWfn06fsPsLWWRJ98vlEJHIyJbqwQvPjscJr90Y/KRQSta+r4XnTPqM6WHlvr5zAJYHL7w1tLKCE5TY2Hee9N5ciPcDUBov7Pv/YtDb2trdIgu8RFUOrFLJr12r1qYRPXX3xzsRKgf5s5gQpYgL781lIaM6Kjip5pyIJ11brDNGfuT/TH342mLh0i1KL36vWH6WOWJZYia5wJpXpUoypqCHMG4VEQoEg9WWP5/TPDeKYtUSCeeHAA/f0vE9Wsm8aCqJC1UGghGHyvPj+KFi2Jp6TT6TSXO9bvnx5O//zrNDX7nEpKU/cPZ5Ph+afuoHUbj/OGgMSbhFgV1XPa5U8+30TTX/1aDRa8yAYca7llZwJNfmK+mjmd8SBNu2zTpg1FRUVRTExMhR+cDwsLY5Z+6rynZ/XHeyKi4VzqFXr372tZ0O2kkYPjlPmGd/iH6SNYK86hhSwUn3zoNpp0b0/64B/rqUmYP8fjpinXzrpNx1SsbOOghnx8hU4mptE/mOXk+3qpe6DNQRvLYI0N146cuMBaznkVu3zh4lVqG9PEGWidXsbDPBls3ZVIDz27QEV0RLUIpqEDY6lX9yh64alh9K//bFF99V9/e4C1so6UfDaTLvCE++GnG+g5NnE/eHuissoQT4xyIDiee2II3T2yE8355Cel2Tu90k4oEBM7NMi5706mhR8/TMFBfspK3LEniabwWIBmOXZUF27PIcL7w4Tx4vThlHA6TV3ff/gszeCJB5t6sKvqDZ6MQ9jKHH1HB3r2pf/SK39eqqyYEUPiaOLYHrRo8S62cg/Q9P8dQq05vtuWHYCONNNxVciOpwfz4NiemHT9Tsw0ySkZahaCUBjYJ0ZpLBC0ew+eVdpTy4jGSugGBvgoEyeFYykRitCtUwtluj3/5FClwVzh7Zfx+87QQV7EiYoMZvMPmlk16/PXW1Lxw7OPDaLnWIj+ZsY3VMwaxGru9HsOptBvua7RXDdnJTDLyMhQZjsEqbk2is/+/v4UEBDAnLy5w9q+nc3e+uGZ0LLbtAplP1g7ZdbDbI1oFqjcKtDIMGHCvEXeWLYiGrNmjncJzTV+32l6dEp/HhikNJE0FrrQbiFcEauJcqHJt+X7vuIJtXwjRBktZM20fZumyty3t+7VeV9n1rAWfPgQHThylt54b4USBNjE4OXJrh6eZM+ev0wDerdW9b+9Xxua99kmnijYX+pWRgP5vC9r6xgTF3hiOcfrCNBKv1q6m8dBKbVt3ZS1urrpO0b/xKS3gC1EjPPs3AJlNR1m8z6ZrQ78DQzAltMy5bLAO45gM33pin08MV6ktT8fVQpUIxaiF9Oy1Q7BZ3hswYTH9aMnUllhOkZPPzpImfmDeWLCNqzdbOEi9hpygqrREK0VTfTe0V0Iwu7t91fTj78cVUJm5pyVFMbmekhjP3rv4x/V+UWL4xVsCNlC9nthwOEzOkuzpgEUEtyQPl24mbDTYsqT/1YAO7ZvRp3jmivTBjNXHJv0+by3v7pnI0uDD5MFZt8S9mXtPnCGYF68wBryuDFdVbss3WP0HDpoZmYmJSUl8fcYZCthCoFq/gOfKK7BL1oTQhSDG513HO9mg18UCf5szXcJN0so+7BHDo2j3qyFhbHfC76tsaM703zWxtA3cL6ALRD4vGNahVCL5oFqcoXJhz4CwTmwbwz9e9FWDpaPop5dI+n/vttFA/q2NoqwxvLPW7CJXp+9XGmh3l6eSnuC7/Zk4kWlSUNQQov6ccMxpZ3D7RMZEazaO5vHxJqfj9Di5Xu5vm5qMgpmDtDgOrFwjmA+obzTrS4m7qLKpQZLDO4taJPYn451AriWEIsMdxs28WBSyWJNezu7MB6d2p8emzaAXnj6DnWcw30BZSAgfhtr4i1Zk394Sj/604t30tfzH1euDbgNftlykpWoc2pBMpHdB9VtznvMmDHjDaPgUSmYXnjB9mwHhHAbPCCWZ4rTtGvvaYpuGUIvPjNcbbPq0z2aBU4KnWIzrU+PKNY6wqgdaxdwJPfmaxCi6Tw7D+7fhgawxrqZ/Wr7eBX4fl6Y6tszmhePWtCm7QlcbrLSguAqgKO+VVQIwXwymvL4SyegIcREhyizw/x+CLDjCWksqMOVU9v8mvYZix0jhrSnSzyDPvXIILprRCftksW/iFTgiViZs2irpWTu00QdUlNTlZDEZ/hG8/LyKv2BWe/lVb4ogbLhwMd9+DFPeMfnU7O483tVWLQzz1PZZ3xhCeoOrUlL6Nw8C6pzjQMbUgzvzIJP+wyznXxvD7X4AGGKxYAR7I7BZFjE5eTxfSPZTIMWByECrWPsnV2UyR7EVglcBGN5Uo5kSwULDZPYnLOnT8IlENs6RKuu+uuRv6TCsbv/pArHRg9iWzehUzyod8Yn0wT28w9j3y4WRGGeww/8yNR+zI2Uxg0+sa3DaNCANtSfhes+tsiwHRETCEKBpo7rRc14AeV7NlszeXFpyrie1JRdIo4kLGQlp6RTbNjGCsWU+IxTgu7nzSdZiz6vFgUrZLBywFVV9zYJbaTGeQf2gyLqApri8VMXCRE6WAg6z2Y8PkOgwlWBaJrm4UHUr3crNcG888EaJXixvoHrMPuhiY5ikx7W6WeLttHm7aeUOwi+0vAmgfTldzvp+MmLhoUo3AtjhnfgRc2bx7ylprrxooTlkWop96/nMIOsWneEhVisMsOsZK33lyCAt8cn0Kih7ZRJYt4gOMyXrT3CC10376nGwIDpAeGEfNCiIZiwwqqZuzBv9cILpqyHexl1aNdU5TV/Hu7DD8KWtIT7IQgTExPVNfhAkcfWhBhSlKGvBzpz/L4U/sKNhtSxXbitxdXLfMCFhc67hrdjdjeaoP9KOFPEdzcu1tAnvEuY/tDeB/aOoddmLVPf0wANzdnpalYBRwscozEdZ1YoGv8exIe1P+ybX/jNLsOTFNoAAa0Sz9VwX8Cd487jARopdkIhubMmWnytVLnnMAkjQgeRGYjIgMyB4IWVo0VewEXAHVcrVglg83O4gG+dMppQ17mzJ9LwwW1v1NtKITdGo5VMcsk4gZ951XUWz57q5fPteNfaAMVMB80JCwj2aE3mtUEHhWYaHR2ttFEjAtS8HPlcNwlgchvFmjgWKb9fdUBNaFhgq08JbYAZbp60cYExYb612MO73MMIxUMLd/PUHJo8hjQBirK0MiyVa36uuj+LEK0mwqPY36f5A/WPgKDz4dVKRwWoVq4mSAMDAw1podr98rduE+jHger4gZZlSXDU7dq7fu1EiFbTO0ZnV4HA1VS+pWJFC7VExXXOiQCtm+/SodV5vR+tbjbRsVr96nKxWogteawWYHYRZWlmv9npWvmIejizbbXSCBseeiu00QYMVb7rW2WShlvcSFvtE6I8urBtL5e3orl6wnY7T0/GZEGyYRLBSnphgfPi83JyEcZhsvS4akNtaTJEJ0I9svnb/V09IfKDXzALEX6Zt3BCOBkWeiwmliwBvN3WiHCxWE4dP4lhDgbYXqviS22or13mfDH/q4x2bZrwSt4JunNYR5ddoce+5E0cNnF73+hK40zxrzxWcfwnwnAacjC0vQmrkQm8C+vshUzq0bnjTeFUKBeDHJ0Yq/HmYU72PtNazCgWv9C279ccpnAOncFCmCvKGIRF/cSB2lEtglQURWVhZWCMLwoJDa6bsZj29gHtPvyPsPWbj1O/Hi20UxX+4tuVxvC+9iUr9pfvU8dOCBdMiCfHLrLOcc34G6U43syGZFeIE8pFuE75boNUJWCw+d+VEmLxsH0R4T0tI4IsCrVyDm684yKVEk9nqvg9ezhw4BIrQm68I8WTA8dbqC/ksDaYIUSdlawJY6yQYnDt4n+dga23Slurpd1fzmqveTl4x5iYIEA7tQ9Xsa3m1/UhTsv2/1FtVuCAMPNs9f4zOGA8x8U2oVY8cXqkTavQJoQ4ISEg/hjHXX63fB9hF5krae4w4dG/2/GW4fs5hhf/GgSKhC3JbiGKwgEeydaHqcz16Jet7Sv/RvDy3VSONE99iz5r+ejUdSVBkOLH2n88rSt1tace1tqnF6IloV+6cF/HxOCm2qdvtyZEwRd9FP3dVRNGHmJRrSkx+rbbZc5rhbiq8DTaPmdtKa2LggqdyUiH0tjVl79G2leW/S25qvjQpu2q2oc+Whf7aW32N4eEaG1WXJ4tBGqagKmg4jbQmn6+PK9uEqhq4qmbtZZaCQEhIATqCAERonXkRUg1hIAQqJ8ExJyvn+9Nal0DBIob3FcDT5FH1HcCDq3O1/fGS/2FgBAQAo4SEHPeUYJyvxAQArc0ARGit/Trl8YLASHgKAGTK+06cBSG3C8EhIAQAAEj3xFgysy8ItSEgBAQAkJAI8Bbgf38fPlb8T1tEqYm7wYNtFvlrxAQAkLglieQlZXFX5juja/ct4mFKSS4/H+/25RbMgkBISAEXJwA/tGjkSQLS0ZoSV4hIASEgI6ACFEdEDkUAkJACBghIELUCC3JKwSEgBDQERAhqgMih0JACAgBIwREiBqhJXmFgBAQAjoCIkR1QORQCAgBIWCEgAhRI7QkrxAQAkJAR0CEqA6IHAoBISAEjBAQIWqEluQVAkJACOgIiBDVAZFDISAEhIARAiJEjdCSvEJACAgBHQERojogcigEhIAQMEJAhKgRWpJXCAgBIaAjIEJUB0QOhYAQEAJGCIgQNUJL8goBISAEdAREiOqAyKEQEAJCwAgBEaJGaEleISAEhICOgAhRHRA5FAJCQAgYISBC1AgtySsEhIAQ0BEQIaoDIodCQAgIASMERIgaoSV5hYAQEAI6AiJEdUDkUAgIASFghIAIUSO0JK8QEAJCQEdAhKgOiBwKASEgBIwQECFqhJbkFQJCQAjoCIgQ1QGRQyEgBISAEQIiRI3QkrxCQAgIAR0BEaI6IHIoBISAEDBCQISoEVqSVwgIASGgI/D/bqy4O7pwVp4AAAAASUVORK5CYII=) ### Preview policy impact[​](#preview-policy-impact) Before enforcing a staged policy, it is a best practice to use the **Preview** feature to avoid unintentionally exposing or blocking other network traffic. 1. From the **Policies Board**, select a staged policy and click **Edit policy**. 2. Make some edits and click **Preview**. ![policy-preview](https://docs.tigera.io/assets/images/policy-preview-b2efca933be175d85ece43ac331086e2.png) ### Enforce a staged policy[​](#enforce-a-staged-policy) 1. From **Policies Board**, click a staged policy. 2. Click**Edit policy**, make changes and click **Enforce**. The staged policy is deleted and the enforced policy is created/updated (depending on whether it already exists). ### Stage updates to an enforced policy[​](#stage-updates-to-an-enforced-policy) 1. From the **Policies Board**, open an enforced policy. 2. In **View Policy**, click **Edit policy**. 3. Make your changes, and click **Preview**. Depending on the results, you can click **Stage** or **Enforce**. You can also use custom resources to stage Kubernetes and Calico Cloud policies, and apply them using `kubectl`. Here are sample YAML files. **Example: StagedGlobalNetworkPolicy** ```yaml apiVersion: projectcalico.org/v3 kind: StagedGlobalNetworkPolicy metadata: name: default.allow-tcp-6379 spec: tier: default selector: role == 'database' types: - Ingress - Egress ingress: - action: Allow protocol: TCP source: selector: role == 'frontend' destination: ports: - 6379 egress: - action: Allow ``` **Example: StagedNetworkPolicy** ```yaml apiVersion: projectcalico.org/v3 kind: StagedNetworkPolicy metadata: name: default.allow-tcp-6379 namespace: default spec: tier: default selector: role == 'database' types: - Ingress - Egress ingress: - action: Allow protocol: TCP source: selector: role == 'frontend' destination: ports: - 6379 egress: - action: Allow ``` **Example: StagedKubernetesNetworkPolicy** ```yaml apiVersion: projectcalico.org/v3 kind: StagedKubernetesNetworkPolicy metadata: name: test-network-policy namespace: default spec: podSelector: matchLabels: role: db policyTypes: - Ingress - Egress ingress: - from: - ipBlock: cidr: 172.17.0.0/16 except: - 172.17.1.0/24 - namespaceSelector: matchLabels: project: myproject - podSelector: matchLabels: role: frontend ports: - protocol: TCP port: 6379 egress: - to: - ipBlock: cidr: 10.0.0.0/24 ports: - protocol: TCP port: 5978 ``` ## Additional resources[​](#additional-resources) - [Staged global network policy](https://docs.tigera.io/calico-cloud/reference/resources/stagedglobalnetworkpolicy) - [Staged network policy](https://docs.tigera.io/calico-cloud/reference/resources/stagednetworkpolicy) - [Staged Kubernetes network policy](https://docs.tigera.io/calico-cloud/reference/resources/stagedkubernetesnetworkpolicy) - For details on how to configure RBAC for staged policy resources, see [Configuring RBAC for tiered policy](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/rbac-tiered-policies) - For details on staged policy metrics, see - [Flow logs](https://docs.tigera.io/calico-cloud/observability/elastic/flow/datatypes) - [Prometheus metrics](https://docs.tigera.io/calico-cloud/operations/monitor/metrics/#content-main) ### Troubleshoot policies ### Problem[​](#problem) I created ​​my first egress policy with default deny behavior, but now I’ve blocked other traffic. ### Solution[​](#solution) In Kubernetes, when there are no egress policies that apply to an endpoint, all egress traffic is allowed. However, as soon as you add the first egress policy to an endpoint, Calico Cloud switches to default deny and blocks everything else; this is part of our zero trust network policy model. For new users of Calico Cloud, this is unexpected behavior (but it’s required by both Kubernetes and Calico Cloud policy specs.) For egress policy in particular, you may not be used to worrying about “system-level” egress traffic that is now suddenly blocked. For example, most workloads rely on DNS, but you may not have thought of this when writing your policy. So you end up with this problem loop: you allow HTTP traffic, but then your DNS traffic gets blocked, but then HTTP traffic stops working because it relies on DNS to function. A natural response to this issue is to add an egress rule to allow DNS(!). For example, you add an egress rule “allow UDP to port 53 to namespace kube-system”. In some systems (OpenShift), the DNS pod actually listens on port 5353, not port 53. However, the DNS Service DNATs the traffic from port 53 to port 5353, hiding that detail from the DNS client. Calico Cloud then blocks the traffic because it sees the traffic after the DNAT. So Calico Cloud sees port 5353, not the expected port 53. The solution is to define policy for workload services, not for ports used by workloads. For help, see [Policy for services](https://docs.tigera.io/calico-cloud/network-policy/beginners/services/). ### Problem[​](#problem-1) Traffic is blocked, even though I allow it in a policy. ### Solution[​](#solution-1) The problem of blocking traffic can reside in your tier, or a different tier. 1. **Check policies in your tier** Go to your policy and see if there is a higher precedent policy in the tier that is blocking processing. - If that is not the problem, go to step 2. - If that is the problem, and if it makes sense for the traffic, you can reorder the policies in the tier. If you cannot, you must change the policy that is dropping traffic to allow your traffic flow using a Pass action rule. 2. **Check policies in other tiers** Go to the next applicable higher precedent tier for your workload to see if a policy in that tier is blocking traffic. The policy at the end of the tier could be blocking traffic because the default behavior at the end of a tier is to drop traffic as part of zero trust. To unblock traffic, add a **Pass action rule** to the policy, or create a **Pass policy**. For help with visibility, use Service Graph to see how traffic is passed. Click on your flow, and view details in the right panel. For help with Pass action rules, see [Get started with tiered policy](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/tiered-policy). ### Calico Cloud network policy for beginners ## [📄️ Enable a default deny policy for Kubernetes pods](https://docs.tigera.io/calico-cloud/network-policy/beginners/kubernetes-default-deny) [Create a default deny network policy so pods that are missing policy are not allowed traffic until appropriate network policy is defined.](https://docs.tigera.io/calico-cloud/network-policy/beginners/kubernetes-default-deny) ## [📄️ Get started with Calico network policy](https://docs.tigera.io/calico-cloud/network-policy/beginners/calico-network-policy) [Create your first Calico network policies. Shows the rich features using sample policies that extend native Kubernetes network policy.](https://docs.tigera.io/calico-cloud/network-policy/beginners/calico-network-policy) ## [📄️ Calico Cloud automatic labels](https://docs.tigera.io/calico-cloud/network-policy/beginners/calico-labels) [Calico Cloud automatic labels for use with resources.](https://docs.tigera.io/calico-cloud/network-policy/beginners/calico-labels) ## [📄️ Calico Cloud for Kubernetes demo](https://docs.tigera.io/calico-cloud/network-policy/beginners/simple-policy-cnx) [Learn the extra features for Calico Cloud that make it so important for production environments.](https://docs.tigera.io/calico-cloud/network-policy/beginners/simple-policy-cnx) ## [🗃️ Policy rules](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/) [6 items](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/) ## [🗃️ Policy for services](https://docs.tigera.io/calico-cloud/network-policy/beginners/services/) [2 items](https://docs.tigera.io/calico-cloud/network-policy/beginners/services/) ### Enable a default deny policy for Kubernetes pods ## Big picture[​](#big-picture) Enable a default deny policy for Kubernetes pods using Kubernetes or Calico Cloud network policy. ## Value[​](#value) A **default deny** network policy provides an enhanced security posture so pods without policy (or incorrect policy) are not allowed traffic until appropriate network policy is defined. ## Features[​](#features) This how-to guide uses the following Calico Cloud features: - **NetworkPolicy** - **GlobalNetworkPolicy** ## Concepts[​](#concepts) ### Default deny/allow behavior[​](#default-denyallow-behavior) **Default allow** means all traffic is allowed by default, unless otherwise specified. **Default deny** means all traffic is denied by default, unless explicitly allowed. **Kubernetes pods are default allow**, unless network policy is defined to specify otherwise. For compatibility with Kubernetes, **Calico Cloud network policy** enforcement follows the standard convention for Kubernetes pods: - If no network policies apply to a pod, then all traffic to/from that pod is allowed. - If one or more network policies apply to a pod with type ingress, then only the ingress traffic specifically allowed by those policies is allowed. - If one or more network policies apply to a pod with type egress, then only the egress traffic specifically allowed by those policies is allowed. For other endpoint types (VMs, host interfaces), the default behavior is to deny traffic. Only traffic specifically allowed by network policy is allowed, even if no network policies apply to the endpoint. ## How to[​](#how-to) - [Create a default deny network policy](#crate-a-default-deny-network-policy) - [Create a global default deny network policy](#create-a-global-default-deny-network-policy) ### Create a default deny network policy[​](#create-a-default-deny-network-policy) Immediately after installation, a best practice is to create a namespaced default deny network policy to secure pods without policy or incorrect policy until you can put policies in place and test them. In the following example, we create a Calico Cloud default deny **NetworkPolicy** for all workloads in the namespace, **engineering**. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: default-deny namespace: engineering spec: selector: all() types: - Ingress - Egress ``` Here's an equivalent default deny **Kubernetes network policy** for all pods in the namespace, **engineering** ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny namespace: engineering spec: podSelector: {} policyTypes: - Ingress - Egress ``` ### Create a global default deny policy[​](#create-a-global-default-deny-policy) A default deny policy ensures that unwanted traffic (ingress and egress) is denied by default without you having to remember default deny/allow behavior of Kubernetes and Calico Cloud policies. This policy can also help mitigate risks of lateral malicious attacks. #### Best practice #1: Allow, stage, then deny[​](#best-practice-1-allow-stage-then-deny) We recommend that you create a global default deny policy after you complete writing policy for the traffic that you want to allow. The following steps summarizes the best practice to test and lock down the cluster to block unwanted traffic: 1. Create a global default deny policy and test it in a staging environment. (The policy will show all the traffic that would be blocked if it were converted into a deny.) 2. Create network policies to individually allow the traffic shown as blocked in step 1 until no connections are denied. 3. Enforce the global default deny policy. #### Best practice #2: Keep the scope to non-system pods[​](#best-practice-2-keep-the-scope-to-non-system-pods) A global default deny policy applies to the entire cluster including all workloads in all namespaces, hosts (computers that run the hypervisor for VMs or container runtime for containers), including Kubernetes control plane and Calico Cloud control plane nodes and pods. For this reason, the best practice is to create a global default deny policy for **non-system pods** as shown in the following example. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: deny-app-policy spec: namespaceSelector: kubernetes.io/metadata.name not in {"calico-system", "kube-public", "kube-system", "tigera-operator"} types: - Ingress - Egress egress: # allow all namespaces to communicate to DNS pods - action: Allow protocol: UDP destination: selector: 'k8s-app == "kube-dns"' ports: - 53 - action: Allow protocol: TCP destination: selector: 'k8s-app == "kube-dns"' ports: - 53 ``` Note the following: - Even though we call this policy "global default deny", the above policy is not explicitly denying traffic. By selecting the traffic with the `namespaceSelector` but not specifying an allow, the traffic is denied after all other policy is evaluated. This design also makes it unnecessary to ensure any specific order (priority) for the default-deny policy. - Allowing access to `kube-dns` simplifies per-pod policies because you don't need to duplicate the DNS rules in every policy - This policy uses a negative selector for the `spec.namespaceselector` field to exclude control plane namespaces. In this example, those namespaces include the `calico-system`, `kube-public`, `kube-system`, and `tigera-operator` namespaces. Because your installation may have different components, make sure to check what you have installed before implementing a similar policy. In a staging environment, verify that the policy does not block any necessary traffic before enforcing it. ### Don't try this\![​](#dont-try-this) The following policy works and looks fine on the surface. But as described in Best practices #2, the policy is too broad in scope and could break your cluster. Therefore, we do not recommend adding this type of policy, even if you have verified allowed traffic in your staging environment. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: default.default-deny spec: tier: default selector: all() types: - Ingress - Egress ``` ## Additional resources[​](#additional-resources) - [Network policy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) - [Global network policy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) ### Get started with Calico network policy ## Big picture[​](#big-picture) Enforce which network traffic that is allowed or denied using rules in Calico network policy. ## Value[​](#value) ### Extends Kubernetes network policy[​](#extends-kubernetes-network-policy) Calico network policy provides a richer set of policy capabilities than Kubernetes including: policy ordering/priority, deny rules, and more flexible match rules. While Kubernetes network policy applies only to pods, Calico network policy can be applied to multiple types of endpoints including pods, VMs, and host interfaces. ### Write once, works everywhere[​](#write-once-works-everywhere) No matter which cloud provider you use now, adopting Calico network policy means you write the policy once and it is portable. If you move to a different cloud provider, you don’t need to rewrite your Calico network policy. Calico network policy is a key feature to avoid cloud provider lock-in. ### Works seamlessly with Kubernetes network policies[​](#works-seamlessly-with-kubernetes-network-policies) You can use Calico network policy in addition to Kubernetes network policy, or exclusively. For example, you could allow developers to define Kubernetes network policy for their microservices. For broader and higher-level access controls that developers cannot override, you could allow only security or Ops teams to define Calico network policies. ## Concepts[​](#concepts) ### Endpoints[​](#endpoints) Calico network policies apply to **endpoints**. In Kubernetes, each pod is a Calico endpoint. However, Calico can support other kinds of endpoints. There are two types of Calico endpoints: **workload endpoints** (such as a Kubernetes pod or OpenStack VM) and **host endpoints** (an interface or group of interfaces on a host). ### Namespaced and global network policies[​](#namespaced-and-global-network-policies) **Calico network policy** is a namespaced resource that applies to pods/containers/VMs in that namespace. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: allow-tcp-port-6379 namespace: production ``` **Calico global network policy** is a non-namespaced resource and can be applied to any kind of endpoint (pods, VMs, host interfaces) independent of namespace. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: allow-tcp-port-6379 ``` Because global network policies use **kind: GlobalNetworkPolicy**, they are grouped separately from **kind: NetworkPolicy**. For example, global network policies will not be returned from `kubectl get networkpolicy.p`, and are rather returned from `kubectl get globalnetworkpolicy`. ### Ingress and egress[​](#ingress-and-egress) Each network policy rule applies to either **ingress** or **egress** traffic. From the point of view of an endpoint (pod, VM, host interface), **ingress** is incoming traffic to the endpoint, and **egress** is outgoing traffic from the endpoint. In a Calico network policy, you create ingress and egress rules independently (egress, ingress, or both). You can specify whether policy applies to ingress, egress, or both using the **types** field. If you do not use the types field, Calico defaults to the following values. | Ingress rule present? | Egress rule present? | Value | | --------------------- | -------------------- | --------------- | | No | No | Ingress | | Yes | No | Ingress | | No | Yes | Egress | | Yes | Yes | Ingress, Egress | ### Network traffic behaviors: deny and allow[​](#network-traffic-behaviors-deny-and-allow) The Kubernetes network policy specification defines the following behavior: - If no network policies apply to a pod, then all traffic to/from that pod is allowed. - If one or more network policies apply to a pod containing ingress rules, then only the ingress traffic specifically allowed by those policies is allowed. - If one or more network policies apply to a pod containing egress rules, then only the egress traffic specifically allowed by those policies is allowed. For compatibility with Kubernetes, **Calico network policy** follows the same behavior for Kubernetes pods. For other endpoint types (VMs, host interfaces), Calico network policy is default deny. That is, only traffic specifically allowed by network policy is allowed, even if no network policies apply to the endpoint. ## How to[​](#how-to) - [Control traffic to/from endpoints in a namespace](#control-traffic-tofrom-endpoints-in-a-namespace) - [Control traffic to/from endpoints independent of namespace](#control-traffic-tofrom-endpoints-independent-of-namespace) - [Control traffic to/from endpoints using IP addresses or CIDR ranges](#control-traffic-tofrom-endpoints-using-ip-addresses-or-cidr-ranges) - [Apply network policies in specific order](#apply-network-policies-in-specific-order) - [Generate logs for specific traffic](#generate-logs-for-specific-traffic) ### Control traffic to/from endpoints in a namespace[​](#control-traffic-tofrom-endpoints-in-a-namespace) In the following example, ingress traffic to endpoints in the **namespace: production** with label **color: red** is allowed, only if it comes from a pod in the same namespace with **color: blue**, on port **6379**. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: allow-tcp-port-6379 namespace: production spec: selector: color == 'red' ingress: - action: Allow protocol: TCP source: selector: color == 'blue' destination: ports: - 6379 ``` To allow ingress traffic from endpoints in other namespaces, use a **namespaceSelector** in the policy rule. A namespaceSelector matches namespaces based on the labels that are applied in the namespace. In the following example, ingress traffic is allowed from endpoints in namespaces that match **shape == circle**. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: allow-tcp-port-6379 namespace: production spec: selector: color == 'red' ingress: - action: Allow protocol: TCP source: selector: color == 'blue' namespaceSelector: shape == 'circle' destination: ports: - 6379 ``` ### Control traffic to/from endpoints independent of namespace[​](#control-traffic-tofrom-endpoints-independent-of-namespace) The following Calico network policy is similar to the previous example, but uses **kind: GlobalNetworkPolicy** so it applies to all endpoints, regardless of namespace. In the following example, incoming TCP traffic to any pods with label **color: red** is denied if it comes from a pod with **color: blue**. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: deny-blue spec: selector: color == 'red' ingress: - action: Deny protocol: TCP source: selector: color == 'blue' ``` As with **kind: NetworkPolicy**, you can allow or deny ingress traffic from endpoints in specific namespaces using a namespaceSelector in the policy rule: ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: deny-circle-blue spec: selector: color == 'red' ingress: - action: Deny protocol: TCP source: selector: color == 'blue' namespaceSelector: shape == 'circle' ``` ### Control traffic to/from endpoints using IP addresses or CIDR ranges[​](#control-traffic-tofrom-endpoints-using-ip-addresses-or-cidr-ranges) Instead of using a selector to define which traffic is allowed to/from the endpoints in a network policy, you can also specify an IP block in CIDR notation. In the following example, outgoing traffic is allowed from pods with the label **color: red** if it goes to an IP address in the **1.2.3.4/24** CIDR block. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: allow-egress-external namespace: production spec: selector: color == 'red' types: - Egress egress: - action: Allow destination: nets: - 1.2.3.0/24 ``` ### Apply network policies in specific order[​](#apply-network-policies-in-specific-order) To control the order/sequence of applying network policies, you can use the **order** field (with precedence from the lowest value to highest). Defining policy **order** is important when you include both **action: allow** and **action: deny** rules that may apply to the same endpoint. In the following example, the policy **allow-cluster-internal-ingress** (order: 10) will be applied before the **policy drop-other-ingress** (order: 20). ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: drop-other-ingress spec: order: 20 #...deny policy rules here... ``` ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: allow-cluster-internal-ingress spec: order: 10 #...allow policy rules here... ``` ### Generate logs for specific traffic[​](#generate-logs-for-specific-traffic) In the following example, incoming TCP traffic to an application is denied, and each connection attempt is logged to syslog. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: allow-tcp-port-6379 namespace: production spec: selector: role == 'database' types: - Ingress - Egress ingress: - action: Log protocol: TCP source: selector: role == 'frontend' - action: Deny protocol: TCP source: selector: role == 'frontend' ``` ### Create policy for established connections[​](#create-policy-for-established-connections) Policies are immediately applied to any new connections. However, for existing connections that are already open, the policy changes will only take effect after the connection has been reestablished. This means that any ongoing sessions may not immediately reflect policy changes until they are initiated again. ## Additional resources[​](#additional-resources) - For additional Calico network policy features, see [Calico network policy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) and [Calico global network policy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) - For an alternative to using IP addresses or CIDRs in policy, see [Network sets](https://docs.tigera.io/calico-cloud/reference/resources/networkset) - For details on how to stage network policy, see [Staged network policies](https://docs.tigera.io/calico-cloud/network-policy/staged-network-policies) ### Calico Cloud automatic labels As a convenience, Calico Cloud provides immutable labels that are used for specific resources when evaluating selectors in policies. The labels make it easier to match resources in common ways (such as matching a namespace by name). ## Labels for matching namespaces[​](#labels-for-matching-namespaces) The label `projectcalico.org/name` is set to the name of the namespace. This allows for matching namespaces by name when using a `namespaceSelector` field. For example, the following GlobalNetworkPolicy applies to workloads with label, `color: red` in namespaces named, `"foo"` and `"bar"`. The policy allows ingress traffic to port 8080 from all workloads in a third namespace named, `"baz"`: ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: foo-and-bar spec: namespaceSelector: projectcalico.org/name in {"foo", "bar"} selector: color == "red" types: - Ingress ingress: - action: Allow source: namespaceSelector: projectcalico.org/name == "baz" destination: ports: - 8080 ``` Be aware that the default values for `namespaceSelector` for NetworkPolicy and GlobalNetworkPolicy are different. For example: **In a network policy**, ```yaml namespaceSelector: selector: foo == "bar" ``` means "resources in the same namespace as the network policy that matches foo == 'bar'". **In a global network policy**, ```yaml namespaceSelector: selector: foo == "bar" ``` means "resources in any namespace and non-namespaced resources that match foo == 'bar'". Further, ```yaml namespaceSelector: projectcalico.org/name == "some-namespace" selector: foo == "bar" ``` is equivalent to: ```yaml namespaceSelector: selector: (foo == "bar") && (projectcalico.org/namespace == "some-namespace") ``` ### Labels for matching service accounts[​](#labels-for-matching-service-accounts) Similarly, the `projectcalico.org/name` label is applied to ServiceAccounts and allows for matching by name in a `serviceAccountSelector`. ### Kubernetes labels for matching namespaces[​](#kubernetes-labels-for-matching-namespaces) Kubernetes also has [automatic labeling](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/#automatic-labelling), for example `kubernetes.io/metadata.name`. The Kubernetes namespace label serves the same purpose and can be used in the same way as the Calico Cloud label. The `projectcalico.org/name` label predates the automatic Kubernetes label. ## Labels for matching host endpoints[​](#labels-for-matching-host-endpoints) [Automatic HostEndpoints](https://docs.tigera.io/calico-cloud/network-policy/hosts/kubernetes-nodes) use the following label to differentiate them from regular HostEndpoints: - `projectcalico.org/created-by: calico-kube-controllers` ## Use the correct selector with labels in policies[​](#use-the-correct-selector-with-labels-in-policies) Calico Cloud labels must be used with the correct selector or the policy will not work as designed (and there are no error messages in the web console or when applying the YAML). | Calico label | Usage requirements | Use in these resources... | | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `projectcalico.org/name` | Use with a **namespaceSelector** or **serviceAccountSelector**. | - Network policy - Staged network policy Namespaced resources that apply only to workload endpoint resources in the namespace. | | `projectcalico.org/namespace` | Use only with selectors. Use the label as the label name, and a namespace name as the value to compare against (for example projectcalico.org/namespace == "default"). | - Global network policy - Staged global network policy Cluster-wide (non-namespaced) resources that apply to workload endpoint resources in all namespaces, and to host endpoint resources. | ### Calico Cloud for Kubernetes demo This guide is a variation of the [simple policy demo](https://docs.tigera.io/calico-cloud/tutorials/kubernetes-tutorials/kubernetes-policy-basic) intended to introduce the extra features of Calico Cloud to people already familiar with Project Calico for Kubernetes. It requires a Kubernetes cluster configured with Calico networking and Calico Cloud, and expects that you have `kubectl` configured to interact with the cluster. This guide assumes that you have installed all the Calico Cloud components from the guides above and that your cluster consists of the following nodes: - k8s-node1 - k8s-node2 - k8s-master Where you see references to these in the text below, substitute for your actual node names. You can find what nodes are on your cluster with `kubectl get nodes` ## Configure Namespaces[​](#configure-namespaces) This guide will deploy pods in a Kubernetes namespace. Let's create the `Namespace` object for this guide. ```text kubectl create ns policy-demo ``` ## Create demo pods[​](#create-demo-pods) We'll use Kubernetes `Deployment` objects to easily create pods in the namespace. 1. Create some nginx pods in the `policy-demo` namespace. ```shell kubectl create deployment --namespace=policy-demo nginx --image=nginx ``` 2. Expose them through a service. ```shell kubectl expose --namespace=policy-demo deployment nginx --port=80 ``` 3. Ensure the nginx service is accessible. ```shell kubectl run --namespace=policy-demo access --rm -ti --image busybox /bin/sh ``` This should open up a shell session inside the `access` pod, as shown below. ```text Waiting for pod policy-demo/access-472357175-y0m47 to be running, status is Pending, pod ready: false If you don't see a command prompt, try pressing enter. / # ``` 4. From inside the `access` pod, attempt to reach the `nginx` service. ```shell wget -q nginx -O - ``` You should see a response from `nginx`. Great! Our service is accessible. You can exit the pod now. 5. Inspect the network policies using calicoq. The `host` command displays information about the policies for endpoints on a given host. > **SECONDARY:** calicoq complements calicoctl by inspecting the dynamic aspects of Calico Cloud Policy: in particular displaying the endpoints actually affected by policies, and the policies that actually apply to endpoints. ```text DATASTORE_TYPE=kubernetes calicoq host k8s-node1 ``` You should see the following output. ```text Policies and profiles for each endpoint on host "k8s-node1": Workload endpoint k8s/tigera-prometheus.alertmanager-calico-node-alertmanager-0/eth0 Policies: Policy "tigera-prometheus/knp.default.calico-node-alertmanager" (order 1000; selector "(projectcalico.org/orchestrator == 'k8s' && alertmanager == 'calico-node-alertmanager' && app == 'alertmanager') && projectcalico.org/namespace == 'tigera-prometheus'") Policy "tigera-prometheus/knp.default.calico-node-alertmanager-mesh" (order 1000; selector "(projectcalico.org/orchestrator == 'k8s' && alertmanager == 'calico-node-alertmanager' && app == 'alertmanager') && projectcalico.org/namespace == 'tigera-prometheus'") Policy "tigera-prometheus/knp.default.default-deny" (order 1000; selector "(projectcalico.org/orchestrator == 'k8s') && projectcalico.org/namespace == 'tigera-prometheus'") Profiles: Profile "kns.tigera-prometheus" Rule matches: Policy "tigera-prometheus/knp.default.calico-node-alertmanager-mesh" inbound rule 1 source match; selector "(projectcalico.org/namespace == 'tigera-prometheus') && (projectcalico.org/orchestrator == 'k8s' && app in { 'alertmanager' } && alertmanager in { 'calico-node-alertmanager' })" ... Workload endpoint k8s/policy-demo.nginx-8586cf59-5bxvh/eth0 Policies: Profiles: Profile "kns.policy-demo" ``` For each workload endpoint, the `Policies:` section lists the policies that apply to that endpoint, in the order they apply. calicoq displays both Calico Cloud Policies and Kubernetes NetworkPolicies, although this example focuses on the latter. The `Rule matches:` section lists the policies that match that endpoint in their rules, in other words that have rules that deny or allow that endpoint as a packet source or destination. Focusing on the `k8s/tigera-prometheus.alertmanager-calico-node-alertmanager-0/eth0` endpoint: - The first two policies are defined in the monitor-calico.yaml manifest. The selectors here have been translated from the original NetworkPolicies to the Calico Cloud format (note the addition of the namespace test). - The third policy and the following profile are created automatically by the policy controller. 6. Use kubectl to see the detail of any particular policy or profile. For example, for the `kns.policy-demo` profile, which defines default behavior for pods in the `policy-demo` namespace: ```shell kubectl get profile kns.policy-demo -o yaml ``` You should see the following output. ```yaml apiVersion: projectcalico.org/v3 kind: Profile metadata: creationTimestamp: '2022-01-06T21:32:05Z' name: kns.policy-demo resourceVersion: 435026/ uid: 75dd2ed4-d3a6-41ca-a106-db073bfa946a spec: egress: - action: Allow destination: {} source: {} ingress: - action: Allow destination: {} source: {} labelsToApply: pcns.projectcalico.org/name: policy-demo ``` Alternatively, you may also use the Calico Cloud web console to inspect and view information and metrics associated with policies, endpoints, and nodes. ## Enable isolation[​](#enable-isolation) Let's turn on isolation in our policy-demo namespace. Calico Cloud will then prevent connections to pods in this namespace. Running the following command creates a NetworkPolicy which implements a default deny behavior for all pods in the `policy-demo` namespace. ```shell kubectl create -f - < **SECONDARY:** The `NetworkPolicy` spec for [`default-deny`](#enable-isolation) does not come configured with any rules. This policy results in a 'default deny' because of how it is [evaluated](https://docs.tigera.io/calico-cloud/reference/resources/tier#how-policy-is-evaluated). A packet will be dropped if the policies (like default-deny in this case) affecting the endpoint takes no action. You can review the metrics associated with such behavior in the **Implicit Drops** block. To view it, go to the policy page and enable **Implicit Drops** from the **Eye** (next to **Add New Tier** button) dropdown. ## Allow access using a NetworkPolicy[​](#allow-access-using-a-networkpolicy) Now, let's enable access to the nginx service using a NetworkPolicy. This will allow incoming connections from our `access` pod, but not from anywhere else. 1. Create a network policy `access-nginx` with the following contents: ```shell kubectl create -f - < **SECONDARY:** The NetworkPolicy allows traffic from pods with the label `run: access` to pods with the label `app: nginx`. These are the labels automatically added to pods started via `kubectl run` based on the name of the `Deployment`. 2. We should now be able to access the service from the `access` pod. ```shell kubectl run --namespace=policy-demo access --rm -ti --image busybox /bin/sh ``` This should open up a shell session inside the `access` pod, as shown below. ```text If you don't see a command prompt, try pressing enter. / # ``` 3. From inside the busybox pod, attempt to access the service again. ```text wget -q --timeout=5 nginx -O - ``` You should see an HTTP response. 4. Return to the **Dashboard** page and review the **Packets by Policy** bar graph to confirm that `access-nginx` causes the packets to be accepted. Other inspection workflows/options include: filtering through the information presented in policies, endpoints and nodes pages. 5. To set a stream of allowed packets run the following command. ```shell for i in `seq 1 10000`; do (wget -q --timeout=1 nginx -O - & sleep 0.01); done ``` 6. Coming back, however, we still cannot access the service from a pod without the label `run: access`: ```shell kubectl run --namespace=policy-demo cant-access --rm -ti --image busybox /bin/sh ``` This should open up a shell session inside the `cant-access` pod, as shown below. ```text If you don't see a command prompt, try pressing enter. / # ``` 7. From inside the `cant-access` pod, attempt to access the service again. ```shell wget -q --timeout=5 nginx -O - ``` After 5 seconds, you should see the following output. ```text wget: download timed out ``` 8. You can clean up the demo by deleting the demo namespace. ```shell kubectl delete ns policy-demo ``` This was just a simple example of the Kubernetes NetworkPolicy API and how Calico Cloud can secure your Kubernetes cluster. For more information on network policy in Kubernetes, see the [Kubernetes user guide](https://kubernetes.io/docs/concepts/services-networking/network-policies/). ### Policy rules ## [📄️ Basic rules](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/policy-rules-overview) [Define network connectivity for Calico endpoints using policy rules and label selectors.](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/policy-rules-overview) ## [📄️ Use namespace rules in policy](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/namespace-policy) [Use namespaces and namespace selectors in Calico network policy to group or separate resources. Use network policies to allow or deny traffic to/from pods that belong to specific namespaces.](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/namespace-policy) ## [📄️ Use service accounts rules in policy](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/service-accounts) [Use Kubernetes service accounts in policies to validate cryptographic identities and/or manage RBAC controlled high-priority rules across teams.](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/service-accounts) ## [📄️ Use service rules in policy](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/service-policy) [Use Kubernetes Service names in policy rules.](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/service-policy) ## [📄️ Use external IPs or networks rules in policy](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/external-ips-policy) [Limit egress and ingress traffic using IP address either directly within Calico network policy or managed as Calico network sets.](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/external-ips-policy) ## [📄️ Use ICMP/ping rules in policy](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/icmp-ping) [Control where ICMP/ping is used by creating a Calico network policy to allow and deny ICMP/ping messages for workloads and host endpoints.](https://docs.tigera.io/calico-cloud/network-policy/beginners/policy-rules/icmp-ping) ### Basic rules ## Big picture[​](#big-picture) Use Calico policy rules and label selectors that match Calico endpoints (pods, OpenStack VMs, and host interfaces) to define network connectivity. ## Value[​](#value) Using label selectors to identify the endpoints (pods, OpenStack VMs, host interfaces) that a policy applies to, or that should be selected by policy rules, means you can define policy without knowing the IP addresses of the endpoints. This is ideal for handling dynamic workloads with ephemeral IPs (such as Kubernetes pods). ## How to[​](#how-to) Read [Get started with Calico policy](https://docs.tigera.io/calico-cloud/network-policy/beginners/calico-network-policy) and [Kubernetes policy](https://docs.tigera.io/calico-cloud/tutorials/kubernetes-tutorials/kubernetes-network-policy), which cover all the basics of using label selectors in policies to select endpoints the policies apply to, or in policy rules. ## Additional resources[​](#additional-resources) - [Global network policy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) - [Network policy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) ### Use namespace rules in policy ## Big picture[​](#big-picture) Use Calico Cloud network policies to reference pods in other namespaces. ## Value[​](#value) Kubernetes namespaces let you group/separate resources to meet a variety of use cases. For example, you can use namespaces to separate development, production, and QA environments, or allow different teams to use the same cluster. You can use namespace selectors in Calico Cloud network policies to allow or deny traffic to/from pods in specific namespaces. ## How to[​](#how-to) - [Control traffic to/from endpoints in a namespace](#control-traffic-tofrom-endpoints-in-a-namespace) - [Use Kubernetes RBAC to control namespace label assignment](#use-kubernetes-rbac-to-control-namespace-label-assignment) ### Control traffic to/from endpoints in a namespace[​](#control-traffic-tofrom-endpoints-in-a-namespace) In the following example, ingress traffic is allowed to endpoints in the **namespace: production** with label **color: red**, and only from a pod in the same namespace with **color: blue**, on **port 6379**. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: allow-tcp-6379 namespace: production spec: selector: color == 'red' ingress: - action: Allow protocol: TCP source: selector: color == 'blue' destination: ports: - 6379 ``` To allow ingress traffic from endpoints in other namespaces, use a **namespaceSelector** in the policy rule. A namespaceSelector matches one or more namespaces based on the labels that are applied on the namespace. In the following example, ingress traffic is also allowed from endpoints with **color: blue** in namespaces with **shape: circle**. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: allow-tcp-6379 namespace: production spec: selector: color == 'red' ingress: - action: Allow protocol: TCP source: selector: color == 'blue' namespaceSelector: shape == 'circle' destination: ports: - 6379 ``` ### Use Kubernetes RBAC to control namespace label assignment[​](#use-kubernetes-rbac-to-control-namespace-label-assignment) Network policies can be applied to endpoints using selectors that match labels on the endpoint, the endpoint's namespace, or the endpoint's service account. By applying selectors based on the endpoint's namespace, you can use Kubernetes RBAC to control which users can assign labels to namespaces. This allows you to separate groups who can deploy pods from those who can assign labels to namespaces. In the following example, users in the development environment can communicate only with pods that have a namespace labeled, `environment == "development"`. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: restrict-development-access spec: namespaceSelector: 'environment == "development"' ingress: - action: Allow source: namespaceSelector: 'environment == "development"' egress: - action: Allow destination: namespaceSelector: 'environment == "development"' ``` ## Additional resources[​](#additional-resources) - For more network policies, see [Network policy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) - To apply policy to all namespaces, see [Global network policy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) ### Use service accounts rules in policy ## Big picture[​](#big-picture) Use Calico Cloud network policy to allow/deny traffic for Kubernetes service accounts. ## Value[​](#value) Using Calico Cloud network policy, you can leverage Kubernetes service accounts with RBAC for flexible control over how policies are applied in a cluster. For example, the security team can have RBAC permissions to: - Control which service accounts the developer team can use within a namespace - Write high-priority network policies for those service accounts (that the developer team cannot override) The network security team can maintain full control of security, while selectively allowing developer operations where it makes sense. ## Concepts[​](#concepts) ### Use smallest set of permissions required[​](#use-smallest-set-of-permissions-required) Operations on service accounts are controlled by RBAC, so you can grant permissions only to trusted entities (code and/or people) to create, modify, or delete service accounts. To perform any operation in a workload, clients are required to authenticate with the Kubernetes API server. If you do not explicitly assign a service account to a pod, it uses the default ServiceAccount in the namespace. You should not grant broad permissions to the default service account for a namespace. If an application needs access to the Kubernetes API, create separate service accounts with the smallest set of permissions required. ### Service account labels[​](#service-account-labels) Like all other Kubernetes objects, service accounts have labels. You can use labels to create ‘groups’ of service accounts. Calico Cloud network policy lets you select workloads by their service account using: - An exact match on service account name - A service account label selector expression ## Before you begin...[​](#before-you-begin) Configure unique Kubernetes service accounts for your applications. ## How to[​](#how-to) - [Limit ingress traffic for workloads by service account name](#limit-ingress-traffic-for-workloads-by-service-account-name) - [Limit ingress traffic for workloads by service account label](#limit-ingress-traffic-for-workloads-by-service-account-label) - [Use Kubernetes RBAC to control service account label assignment](#use-kubernetes-rbac-to-control-service-account-label-assignment) ### Limit ingress traffic for workloads by service account name[​](#limit-ingress-traffic-for-workloads-by-service-account-name) In the following example, ingress traffic is allowed from any workload whose service account matches the names **api-service** or **user-auth-service**. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: demo-calico namespace: prod-engineering spec: ingress: - action: Allow source: serviceAccounts: names: - api-service - user-auth-service selector: 'app == "db"' ``` ### Limit ingress traffic for workloads by service account label[​](#limit-ingress-traffic-for-workloads-by-service-account-label) In the following example, ingress traffic is allowed from any workload whose service account matches the label selector, **app == web-frontend**. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: allow-web-frontend namespace: prod-engineering spec: ingress: - action: Allow source: serviceAccounts: selector: 'app == "web-frontend"' selector: 'app == "db"' ``` ### Use Kubernetes RBAC to control service account label assignment[​](#use-kubernetes-rbac-to-control-service-account-label-assignment) Network policies can be applied to endpoints using selectors that match labels on the endpoint, the endpoint's namespace, or the endpoint's service account. By applying selectors based on the endpoint's service account, you can use Kubernetes RBAC to control which users can assign labels to service accounts. This allows you to separate groups who can deploy pods from those who can assign labels to service accounts. In the following example, pods with an intern service account can communicate only with pods with service accounts labeled, `role: intern`. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: restrict-intern-access namespace: prod-engineering spec: serviceAccountSelector: 'role == "intern"' ingress: - action: Allow source: serviceAccounts: selector: 'role == "intern"' egress: - action: Allow destination: serviceAccounts: selector: 'role == "intern"' ``` ## Additional resources[​](#additional-resources) - [Network policy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) - [Global network policy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) ### Use service rules in policy ## Big picture[​](#big-picture) Use Calico Cloud network policy to allow/deny traffic for Kubernetes services. ## Value[​](#value) Using Calico Cloud network policy, you can leverage Kubernetes Service names to easily define access to Kubernetes services. Using service names in policy enables you to: - Allow or deny access to the Kubernetes API service. - Reference port information already declared by the application, making it easier to keep policy up-to-date as application requirements change. ## How to[​](#how-to) - [Allow access to the Kubernetes API for a specific namespace](#allow-access-to-the-kubernetes-api-for-a-specific-namespace) - [Allow access to Kubernetes DNS for the entire cluster](#allow-access-to-kubernetes-dns-for-the-entire-cluster) - [Allow access from a specified service](#allow-access-from-a-specified-service) ### Allow access to the Kubernetes API for a specific namespace[​](#allow-access-to-the-kubernetes-api-for-a-specific-namespace) In the following example, egress traffic is allowed to the `kubernetes` service in the `default` namespace for all pods in the namespace `my-app`. This service is the typical access point for the Kubernetes API server. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: allow-api-access namespace: my-app spec: selector: all() egress: - action: Allow destination: services: name: kubernetes namespace: default ``` Endpoint addresses and ports to allow will be automatically detected from the service. ### Allow access to Kubernetes DNS for the entire cluster[​](#allow-access-to-kubernetes-dns-for-the-entire-cluster) In the following example, a GlobalNetworkPolicy is used to select all pods in the cluster to apply a rule which ensures all pods can access the Kubernetes DNS service. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: allow-kube-dns spec: selector: all() egress: - action: Allow destination: services: name: kube-dns namespace: kube-system ``` > **SECONDARY:** This policy also enacts a default-deny behavior for all pods, so make sure any other required application traffic is allowed by a policy. ## Allow access from a specified service[​](#allow-access-from-a-specified-service) In the following example, ingress traffic is allowed from the `frontend-service` service in the `frontend` namespace for all pods in the namespace `backend`. This allows all pods that back the `frontend-service` service to send traffic to all pods in the `backend` namespace. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: allow-frontend-service-access namespace: backend spec: selector: all() ingress: - action: Allow source: services: name: frontend-service namespace: frontend ``` We can also further specify the ports that the `frontend-service` service is allowed to access. The following example limits access from the `frontend-service` service to port 80. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: allow-frontend-service-access namespace: backend spec: selector: all() ingress: - action: Allow protocol: TCP source: services: name: frontend-service namespace: frontend destination: ports: [80] ``` ## Additional resources[​](#additional-resources) - [Network policy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) - [Global network policy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) ### Use external IPs or networks rules in policy ## Big picture[​](#big-picture) Use Calico Cloud network policy to limit traffic to/from external non-Calico Cloud workloads or networks. ## Value[​](#value) Modern applications often integrate with third-party APIs and SaaS services that live outside Kubernetes clusters. To securely enable access to those integrations, network security teams must be able to limit IP ranges for egress and ingress traffic to workloads. This includes using IP lists or ranges to deny-list bad actors or embargoed countries. Using Calico Cloud network policy, you can define IP addresses/CIDRs directly in policy to limit traffic to external networks. Or using Calico Cloud network sets, you can easily scale out by using the same set of IPs in multiple policies. ## Concepts[​](#concepts) ### IP addresses/CIDRs[​](#ip-addressescidrs) IP addresses and CIDRs can be specified directly in both Kubernetes and Calico Cloud network policy rules. Calico Cloud network policy supports IPV4 and IPV6 CIDRs. ### Network sets[​](#network-sets) A **network set** resource is an arbitrary set of IP subnetworks/CIDRs that can be matched by standard label selectors in Kubernetes or Calico Cloud network policy. This is useful to reference a set of IP addresses using a selector from a namespaced network policy resource. It is typically used when you want to scale/reuse the same set of IP addresses in policy. A **global network set** resource is similar, but can be selected only by Calico Cloud global network policies. ## How to[​](#how-to) - [Limit traffic to or from external networks, IPs in network policy](#limit-traffic-to-or-from-external-networks-ips-in-network-policy) - [Limit traffic to or from external networks, global network set](#limit-traffic-to-or-from-external-networks-global-network-set) ### Limit traffic to or from external networks, IPs in network policy[​](#limit-traffic-to-or-from-external-networks-ips-in-network-policy) In the following example, a Calico Cloud NetworkPolicy allows egress traffic from pods with the label **color: red**, if it goes to an IP address in the 192.0.2.0/24 CIDR block. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: allow-egress-external namespace: production spec: selector: color == 'red' types: - Egress egress: - action: Allow destination: nets: - 192.0.2.0/24 ``` ### Limit traffic to or from external networks, global network set[​](#limit-traffic-to-or-from-external-networks-global-network-set) In this example, we use a Calico Cloud **GlobalNetworkSet** and reference it in a **GlobalNetworkPolicy**. In the following example, a Calico Cloud **GlobalNetworkSet** deny-lists the CIDR ranges 192.0.2.55/32 and 203.0.113.0/24: ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkSet metadata: name: ip-protect labels: ip-deny-list: 'true' spec: nets: - 192.0.2.55/32 - 203.0.113.0/24 ``` Next, we create two Calico Cloud **GlobalNetworkPolicy** objects. The first is a high “order” policy that allows traffic as a default for things that don’t match our second policy, which is low “order” and uses the **GlobalNetworkSet** label as a selector to deny ingress traffic (IP-deny-list in the previous step). In the label selector, we also include the term **!has(projectcalico.org/namespace)**, which prevents this policy from matching pods or NetworkSets that also have this label. To more quickly enforce the denial of forwarded traffic to the host at the packet level, use the **doNotTrack** and **applyOnForward** options. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: forward-default-allow spec: selector: apply-ip-protect == 'true' order: 1000 doNotTrack: true applyOnForward: true types: - Ingress ingress: - action: Allow --- apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: ip-protect spec: selector: apply-ip-protect == 'true' order: 0 doNotTrack: true applyOnForward: true types: - Ingress ingress: - action: Deny source: selector: ip-deny-list == 'true' && !has(projectcalico.org/namespace) ``` ## Additional resources[​](#additional-resources) - To understand how to use global network sets to mitigate common threats, see [Defend against DoS attacks](https://docs.tigera.io/calico-cloud/network-policy/extreme-traffic/defend-dos-attack) - [Global network sets](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkset) - [Global network policy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) ### Use ICMP/ping rules in policy ## Big picture[​](#big-picture) Use Calico Cloud network policy to allow and deny ICMP/ping messages. ## Value[​](#value) The **Internet Control Message Protocol (ICMP)** provides valuable network diagnostic functions, but it can also be used maliciously. Attackers can use it to learn about your network, or for DoS attacks. Using Calico Cloud network policy, you can control where ICMP is used. For example, you can: - Allow ICMP ping, but only for workloads, host endpoints (or both) - Allow ICMP for pods launched by operators for diagnostic purposes, but block other uses - Temporarily enable ICMP to diagnose a problem, then disable it after the problem is resolved - Deny/allow ICMPv4 and/or ICMPv6 ## Concepts[​](#concepts) ### ICMP packet type and code[​](#icmp-packet-type-and-code) Calico Cloud network policy also lets you deny and allow ICMP traffic based on specific types and codes. For example, you can specify ICMP type 5, code 2 to match specific ICMP redirect packets. For details, see [ICMP type and code](https://en.wikipedia.org/wiki/Internet_Control_Message_Protocol#Control_messages). ## How to[​](#how-to) - [Deny all ICMP, all workloads and host endpoints](#deny-all-icmp-all-workloads-and-host-endpoints) - [Allow ICMP ping, all workloads and host endpoints](#allow-icmp-ping-all-workloads-and-host-endpoints) - [Allow ICMP matching protocol type and code, all Kubernetes pods](#allow-icmp-matching-protocol-type-and-code-all-kubernetes-pods) ### Deny all ICMP, all workloads and host endpoints[​](#deny-all-icmp-all-workloads-and-host-endpoints) In this example, we introduce a "deny all ICMP" **GlobalNetworkPolicy**. This policy **selects all workloads and host endpoints**. It enables a default deny for all workloads and host endpoints, in addition to the explicit ICMP deny rules specified in the policy. If your ultimate goal is to allow some traffic, have your regular "allow" policies in place before applying a global deny-all ICMP traffic policy. In this example, all workloads and host endpoints are blocked from sending or receiving **ICMPv4** and **ICMPv6** messages. If **ICMPv6** messages are not used in your deployment, it is still good practice to deny them specifically as shown below. In any "deny-all" Calico Cloud network policy, be sure to specify a lower order (**order:200**) than regular policies that might allow traffic. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: block-icmp spec: order: 200 selector: all() types: - Ingress - Egress ingress: - action: Deny protocol: ICMP - action: Deny protocol: ICMPv6 egress: - action: Deny protocol: ICMP - action: Deny protocol: ICMPv6 ``` ### Allow ICMP ping, all workloads and host endpoints[​](#allow-icmp-ping-all-workloads-and-host-endpoints) In this example, workloads and host endpoints can receive **ICMPv4 type 8** and **ICMPv6 type 128** ping requests that come from other workloads and host endpoints. All other traffic may be allowed by other policies. If traffic is not explicitly allowed, it will be denied by default. The policy applies only to **ingress** traffic. (Egress traffic is not affected, and default deny is not enforced for egress.) ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: allow-ping-in-cluster spec: selector: all() types: - Ingress ingress: - action: Allow protocol: ICMP source: selector: all() icmp: type: 8 Ping request - action: Allow protocol: ICMPv6 source: selector: all() icmp: type: 128 Ping request ``` ### Allow ICMP matching protocol type and code, all Kubernetes pods[​](#allow-icmp-matching-protocol-type-and-code-all-kubernetes-pods) In this example, only Kubernetes pods that match the selector **projectcalico.org/orchestrator == 'kubernetes'** are allowed to receive ICMPv4 **code: 1 host unreachable** messages. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: allow-host-unreachable spec: selector: projectcalico.org/orchestrator == 'kubernetes' types: - Ingress ingress: - action: Allow protocol: ICMP icmp: type: 3 Destination unreachable code: 1 Host unreachable ``` ## Additional resources[​](#additional-resources) For more on the ICMP match criteria, see: - [Global network policy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) - [Network policy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) ### Policy for Kubernetes services ## [📄️ Apply Calico Cloud policy to Kubernetes node ports](https://docs.tigera.io/calico-cloud/network-policy/beginners/services/kubernetes-node-ports) [Restrict access to Kubernetes node ports using Calico Cloud global network policy. Follow the steps to secure the host, the node ports, and the cluster.](https://docs.tigera.io/calico-cloud/network-policy/beginners/services/kubernetes-node-ports) ## [📄️ Apply Calico Cloud policy to services exposed externally as cluster IPs](https://docs.tigera.io/calico-cloud/network-policy/beginners/services/services-cluster-ips) [Expose Kubernetes service cluster IPs over BGP using Calico Cloud, and restrict who can access them using Calico Cloud network policy.](https://docs.tigera.io/calico-cloud/network-policy/beginners/services/services-cluster-ips) ### Apply Calico Cloud policy to Kubernetes node ports ## Big picture[​](#big-picture) Restrict access to node ports to specific external clients. ## Value[​](#value) Exposing services to external clients using node ports is a standard Kubernetes feature. However, if you want to restrict access to node ports to specific external clients, you need to use Calico global network policy. ## Concepts[​](#concepts) ### Network policy with preDNAT field[​](#network-policy-with-prednat-field) In a Kubernetes cluster, kube-proxy will DNAT a request to the node's port and IP address to one of the pods that backs the service. For Calico global network policy to both allow normal ingress cluster traffic and deny other general ingress traffic, it must take effect before DNAT. To do this, you simply add a **preDNAT** field to a Calico global network policy. The preDNAT field: - Applies before DNAT - Applies only to ingress rules - Enforces all ingress traffic through a host endpoint, regardless of destination The destination can be a locally hosted pod, a pod on another node, or a process running on the host. ## Before you begin...[​](#before-you-begin) For services that you want to expose to external clients, configure Kubernetes services with type **NodePort**. ## How to[​](#how-to) To securely expose a Kubernetes service to external clients, you must implement all of the following steps. - [Allow cluster ingress traffic, but deny general ingress traffic](#allow-cluster-ingress-traffic-but-deny-general-ingress-traffic) - [Allow local host egress traffic](#allow-local-host-egress-traffic) - [Create host endpoints with appropriate network policy](#create-host-endpoints-with-appropriate-network-policy) - [Allow ingress traffic to specific node ports](#allow-ingress-traffic-to-specific-node-ports) ### Allow cluster ingress traffic but deny general ingress traffic[​](#allow-cluster-ingress-traffic-but-deny-general-ingress-traffic) In the following example, we create a global network policy to allow cluster ingress traffic (**allow-cluster-internal-ingress**): for the nodes’ IP addresses (**1.2.3.4/16**), and for pod IP addresses assigned by Kubernetes (**100.100.100.0/16**). By adding a preDNAT field, Calico global network policy is applied before regular DNAT on the Kubernetes cluster. In this example, we use the **selector: has(kubernetes-host)** -- so the policy is applicable to any endpoint with a **kubernetes-host** label (but you can easily specify particular nodes). Finally, when you specify a preDNAT field, you must also add the **applyOnForward: true** field. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: allow-cluster-internal-ingress-only spec: order: 20 preDNAT: true applyOnForward: true ingress: - action: Allow source: nets: [1.2.3.4/16, 100.100.100.0/16] - action: Deny selector: has(kubernetes-host) ``` ### Allow local host egress traffic[​](#allow-local-host-egress-traffic) We also need a global network policy to allow egress traffic through each node's external interface. Otherwise, when we define host endpoints for those interfaces, no egress traffic will be allowed from local processes (except for traffic that is allowed by the [Failsafe rules](https://docs.tigera.io/calico-cloud/reference/host-endpoints/failsafe). ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: allow-outbound-external spec: order: 10 egress: - action: Allow selector: has(kubernetes-host) ``` ### Create host endpoints with appropriate network policy[​](#create-host-endpoints-with-appropriate-network-policy) In this example, we assume that you have already defined Calico host endpoints with network policy that is appropriate for the cluster. (For example, you wouldn’t want a host endpoint with a “default deny all traffic to/from this host” network policy because that is counter to the goal of allowing/denying specific traffic.) For help, see [host endpoints](https://docs.tigera.io/calico-cloud/reference/resources/hostendpoint). All of our previously-defined global network policies have a selector that makes them applicable to any endpoint with a **kubernetes-host label**; so we will include that label in our definitions. For example, for **eth0** on **node1**. ```yaml apiVersion: projectcalico.org/v3 kind: HostEndpoint metadata: name: node1-eth0 labels: kubernetes-host: ingress spec: interfaceName: eth0 node: node1 expectedIPs: - INSERT_IP_HERE ``` When creating each host endpoint, replace `INSERT_IP_HERE` with the IP address on eth0. The `expectedIPs` field is required so that any selectors within ingress or egress rules can properly match the host endpoint. ### Allow ingress traffic to specific node ports[​](#allow-ingress-traffic-to-specific-node-ports) Now we can allow external access to the node ports by creating a global network policy with the preDNAT field. In this example, **ingress traffic is allowed** for any host endpoint with **port: 31852**. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: allow-nodeport spec: preDNAT: true applyOnForward: true order: 10 ingress: - action: Allow protocol: TCP destination: selector: has(kubernetes-host) ports: [31852] selector: has(kubernetes-host) ``` To make the NodePort accessible only through particular nodes, give the nodes a particular label. For example: ```yaml nodeport-external-ingress: true ``` Then, use **nodeport-external-ingress: true** as the selector of the **allow-nodeport** policy, instead of **has(kubernetes-host)**. ## Additional resources[​](#additional-resources) - [Global network policy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) - [Host endpoints](https://docs.tigera.io/calico-cloud/reference/resources/hostendpoint) ### Apply Calico Cloud policy to services exposed externally as cluster IPs ## Big picture[​](#big-picture) Control access to services exposed through clusterIPs that are advertised outside the cluster using BGP. ## Value[​](#value) Calico Cloud network policy uses standard Kubernetes Services that allow you to expose services within clusters to external clients in the following ways: - [Apply policy to Kubernetes nodeports](https://docs.tigera.io/calico-cloud/network-policy/beginners/services/kubernetes-node-ports) - Using cluster IPs over BGP (described in this article) ## Concepts[​](#concepts) ### Advertise cluster IPs outside the cluster[​](#advertise-cluster-ips-outside-the-cluster) A **cluster IP** is a virtual IP address that represents a Kubernetes Service. Kube Proxy on each host translates the clusterIP into a pod IP for one of the pods backing the service, acting as a reverse proxy and load balancer. Cluster IPs were originally designed for use within the Kubernetes cluster. Calico Cloud allows you to advertise Cluster IPs externally -- so external clients can use them to access services hosted inside the cluster. This means that Calico Cloud ingress policy can be applied at **one or both** of the following locations: - Host interface, when the traffic destined for the clusterIP first ingresses the cluster - Pod interface of the backend pod ### Traffic routing: local versus cluster modes[​](#traffic-routing-local-versus-cluster-modes) Calico Cloud implements [Kubernetes service external traffic policy](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip), which controls whether external traffic is routed to node-local or cluster-wide endpoints. The following table summarizes key differences between these settings. The default is **cluster mode**. | **Service setting** | **Traffic is load balanced...** | **Pros and cons** | **Required service type** | | ------------------------------------------- | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------- | | **externalTrafficPolicy: Cluster**(default) | Across all nodes in the cluster | Equal distribution of traffic among all pods running a service. Possible unnecessary network hops between nodes for ingress external traffic.When packets are rerouted to pods on another node, traffic is SNAT’d (source network address translation). Destination pod can see the proxying node’s IP address rather than the actual client IP. | **ClusterIP** | | **externalTrafficPolicy: Local** | Across the nodes with the endpoints for the service | Avoids extra hops so better for apps that ingress a lot external traffic. Traffic is not SNAT’d so actual client IPs are preserved. Traffic distributed among pods running a service may be imbalanced. | **LoadBalancer** (for cloud providers), or **NodePort** (for node’s static port) | ## Before you begin...[​](#before-you-begin) [Configure Calico to advertise cluster IPs over BGP](https://docs.tigera.io/calico-cloud/networking/configuring/advertise-service-ips). ## How to[​](#how-to) Selecting which mode to use depends on your goals and resources. At an operational level, **local mode** simplifies policy, but load balancing may be uneven in certain scenarios. **Cluster mode** requires more work to manage clusterIPs, SNAT, and create policies that reference specific IP addresses, but you always get even load balancing. - [Secure externally exposed cluster IPs, local mode](#secure-externally-exposed-cluster-ips-local-mode) - [Secure externally exposed cluster IPs, cluster mode](#secure-externally-exposed-cluster-ips-cluster-mode) ### Secure externally exposed cluster IPs, local mode[​](#secure-externally-exposed-cluster-ips-local-mode) Using **local mode**, the original source address of external traffic is preserved, and you can define policy directly using standard Calico Cloud network policy. 1. Create Calico Cloud **NetworkPolicies** or **GlobalNetworkPolicies** that select the same set of pods as your Kubernetes Service. 2. Add rules to allow the external traffic. 3. If desired, add rules to allow in-cluster traffic. ### Secure externally exposed cluster IPs, cluster mode[​](#secure-externally-exposed-cluster-ips-cluster-mode) In the following steps, we define **GlobalNetworkPolicy** and **HostEndpoints**. #### Step 1: Verify Kubernetes Service manifest[​](#step-1-verify-kubernetes-service-manifest) Ensure that your Kubernetes Service manifest explicitly lists the clusterIP; do not allow Kubernetes to automatically assign the clusterIP because you need it for your policies in the following steps. #### Step 2: Create global network policy at the host interface[​](#step-2-create-global-network-policy-at-the-host-interface) In this step, you create a **GlobalNetworkPolicy** that selects all **host endpoints**. It controls access to the cluster IP, and prevents unauthorized clients from outside the cluster from accessing it. The hosts then forwards only authorized traffic. **Set policy to allow external traffic for cluster IPs** Add rules to allow the external traffic for each clusterIP. The following example allows connections to two cluster IPs. Make sure you add **applyOnForward** and **preDNAT** rules. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: allow-cluster-ips spec: selector: k8s-role == 'node' types: - Ingress applyOnForward: true preDNAT: true ingress: # Allow 50.60.0.0/16 to access Cluster IP A - action: Allow source: nets: - 50.60.0.0/16 destination: nets: - 10.20.30.40/32 Cluster IP A # Allow 70.80.90.0/24 to access Cluster IP B - action: Allow source: nets: - 70.80.90.0/24 destination: nets: - 10.20.30.41/32 Cluster IP B ``` **Add a rule to allow traffic destined for the pod CIDR** Without this rule, normal pod-to-pod traffic is blocked because the policy applies to forwarded traffic. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: allow-to-pods spec: selector: k8s-role == 'node' types: - Ingress applyOnForward: true preDNAT: true ingress: # Allow traffic forwarded to pods - action: Allow destination: nets: - 192.168.0.0/16 Pod CIDR ``` **Add a rule to allow traffic destined for all host endpoints** Or, you can add rules that allow specific host traffic including Kubernetes and Calico Cloud. Without this rule, normal host traffic is blocked. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: allow-traffic-hostendpoints spec: selector: k8s-role == 'node' types: - Ingress # Allow traffic to the node (not nodePorts, TCP) (not nodePorts, TCP) - action: Allow protocol: TCP destination: selector: k8s-role == 'node' notPorts: ["30000:32767"] #nodePort range # Allow traffic to the node (not nodePorts, TCP) (not nodePorts, UDP) - action: Allow protocol: UDP destination: selector: k8s-role == 'node' notPorts: ["30000:32767"] #nodePort range ``` #### Step 3: Create a global network policy that selects pods[​](#step-3-create-a-global-network-policy-that-selects-pods) In this step, you create a **GlobalNetworkPolicy** that selects the **same set of pods as your Kubernetes Service**. Add rules that allow host endpoints to access the service ports. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: allow-nodes-svc-a spec: selector: k8s-svc == 'svc-a' types: - Ingress ingress: - action: Allow protocol: TCP source: selector: k8s-role == 'node' destination: ports: [80, 443] - action: Allow protocol: UDP source: selector: k8s-role == 'node' destination: ports: [80, 443] ``` #### Step 4: (Optional) Create network polices or global network policies that allow in-cluster traffic to access the service[​](#step-4-optional-create-network-polices-or-global-network-policies-that-allow-in-cluster-traffic-to-access-the-service) #### Step 5: Create HostEndpoints[​](#step-5-create-hostendpoints) Create HostEndpoints for the interface of each host that will receive traffic for the clusterIPs. Be sure to label them so they are selected by the policy in Step 2 (Add a rule to allow traffic destined for the pod CIDR), and the rules in Step 3. In the previous example policies, the label **k8s-role: node** is used to identify these HostEndpoints. ## Additional resources[​](#additional-resources) - [Enable service IP advertisement](https://docs.tigera.io/calico-cloud/networking/configuring/advertise-service-ips) - [Defend against DoS attacks](https://docs.tigera.io/calico-cloud/network-policy/extreme-traffic/defend-dos-attack) - [Global network policy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) ### DNS policy ## Big picture[​](#big-picture) Use domain names to allow traffic to destinations outside of a cluster. ## Value[​](#value) Using domain names in policies to identify services outside of the cluster is often operationally simpler and more robust than using IP addresses. In particular, they are useful when an external service does not map to a well known set of static IP addresses. Domain names can include a wildcard (`*`), making it easier to manage large numbers of domains/subdomains. ## Concepts[​](#concepts) ### Allowed egress domains[​](#allowed-egress-domains) Using domain names in policy rules is limited to only egress allow rules. Calico Cloud allows connections only to IP addresses returned from DNS lookups to trusted DNS servers. The supported DNS types are: A, AAAA, and CNAME records. The domain name must be an exact match; for example, **google.com** is treated as distinct from **[www.google.com](http://www.google.com)**. > **SECONDARY:** Kubernetes labels provide a similar convenience for services within the cluster. Calico Enterprise does not support using domain names for services within the cluster. Use Kubernetes labels for services within the cluster. ### Domain name matching[​](#domain-name-matching) When a configured domain name has no wildcard (`*`), it matches exactly that domain name. For example: - `microsoft.com` - `tigera.io` With a single asterisk in any part of the domain name, it matches 1 or more path components at that position. For example: - `*.google.com` matches `www.google.com` and `www.ipv6.google.com`, but not `google.com` - `www.*.com` matches `www.sun.com` and `www.apple.com`, but not `www.com` - `update.*.mycompany.com` matches `update.tools.mycompany.com`, `update.secure.suite.mycompany.com`, and so on **Not** supported are: - Multiple wildcards in the same domain, for example: `*.*.mycompany.com` - Asterisks that are not the entire component, for example: `www.g*.com` - A wildcard as the last component, for example: `www.mycompany.*` - More general wildcards, such as regular expressions ### Workload and host endpoints[​](#workload-and-host-endpoints) Policy with domain names can be enforced on workload or host endpoints. When a policy with domain names applies to a workload endpoint, it allows that workload to connect out to the specified domains. When policy with domain names applies to a host endpoint, it allows clients directly on the relevant host (including any host-networked workloads) to connect out to the specified domains. ### Trusted DNS servers[​](#trusted-dns-servers) Calico Cloud trusts DNS information only from its list of DNS trusted servers. Using trusted DNS servers to back domain names in policy, prevents a malicious workload from using IPs returned by a fake DNS server to hijack domain names in policy rules. By default, Calico Cloud trusts the Kubernetes cluster’s DNS service (kube-dns or CoreDNS). For workload endpoints, these out-of-the-box defaults work with standard Kubernetes installs, so normally you won’t change them. For host endpoints you will need to add the IP addresses that the cluster nodes use for DNS resolution. ## Before you begin[​](#before-you-begin) **Not supported** DNS policy is not supported at egress of egress gateway pods. Domain-based rules will either never match in that hook, or, they may match intermittently. Intermittent matches occur when a pod on the same node as the egress gateway pod happens to make a matching DNS query. This is because the DNS-to-IP cache used to render the policy is shared node-wide. ## How to[​](#how-to) You can specify allowed domain names directly in a **global network policy** or **namespaced network policy**, or specify domain names in a **global network set** (and then reference the global network set in a global network policy). - [Use domain names in a global network policy](#use-domain-names-in-a-global-network-policy) - [Use domain names in a namespaced network policy](#use-domain-names-in-a-namespaced-network-policy) - [Use domain names in a global network set, reference the set in a global network policy](#use-domain-names-in-a-global-network-set) ### Best practice[​](#best-practice) Use a **global network set** when the same set of domains needs to be referenced in multiple policies, or when you want the allowed destinations to be a mix of domains and IPs from global network sets, or IPs from workload endpoints and host endpoints. By using a single destination selector in a global network set, you can potentially match all of these resources. ### Use domain names in a global network policy[​](#use-domain-names-in-a-global-network-policy) In this method, you create a **GlobalNetworkPolicy** with egress rules with `action: Allow` and a `destination.domains` field specifying the domain names to which egress traffic is allowed. In the following example, the first rule allows DNS traffic, and the second rule allows connections outside the cluster to domains **api.alice.com** and **\*.example.com** (which means `.example.com`, such as **bob.example.com**). ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: allow-egress-to-domains spec: order: 1 selector: my-pod-label == 'my-value' types: - Egress egress: - action: Allow protocol: UDP destination: ports: - 53 - dns - action: Allow destination: domains: - api.alice.com - '*.example.com' ``` ### Use domain names in a namespaced network policy[​](#use-domain-names-in-a-namespaced-network-policy) In this method, you create a **NetworkPolicy** with egress rules with `action: Allow` and a `destination.domains` field specifying the domain names to which egress traffic is allowed. In the following example, the first rule allows DNS traffic, and the second rule allows connections outside the cluster to domains **api.alice.com** and **\*.example.com** (which means `.example.com`, such as **bob.example.com**). ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: allow-egress-to-domains namespace: rollout-test spec: order: 1 selector: my-pod-label == 'my-value' types: - Egress egress: - action: Allow protocol: UDP destination: ports: - 53 - dns - action: Allow destination: domains: - api.alice.com - '*.example.com' ``` The difference between this and the **GlobalNetworkPolicy** example is that this namespaced NetworkPolicy can only grant egress access, to the specified domains, to workload endpoints in the `rollout-test` namespace. ### Use domain names in a global network set[​](#use-domain-names-in-a-global-network-set) In this method, you create a **GlobalNetworkSet** with the allowed destination domain names in the `allowedEgressDomains` field. Then, you create a **GlobalNetworkPolicy** with a `destination.selector` that matches that GlobalNetworkSet. In the following example, the allowed egress domains (`api.alice.com` and `*.example.com`) are specified in the GlobalNetworkSet. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkSet metadata: name: allowed-domains-1 labels: color: red spec: allowedEgressDomains: - api.alice.com - '*.example.com' ``` Then, you reference the global network set in a **GlobalNetworkPolicy** using a destination label selector. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: allow-egress-to-domain spec: order: 1 selector: my-pod-label == 'my-value' types: - Egress egress: - action: Allow destination: selector: color == 'red' ``` ### Use domain names in a network set[​](#use-domain-names-in-a-network-set) In this method, you create a **NetworkSet** with the allowed destination domain names in the `allowedEgressDomains` field. Then, you create a **NetworkPolicy** with a `destination.selector` that matches that NetworkSet. In the following example, the allowed egress domains (`api.alice.com` and `*.example.com`) are specified in the NetworkSet. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkSet metadata: name: allowed-domains-1 namespace: rollout-test labels: color: red spec: allowedEgressDomains: - api.alice.com - '*.example.com' ``` Then, you reference the network set in a **NetworkPolicy** using a destination label selector. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: allow-egress-to-domain namespace: rollout-test spec: order: 1 selector: my-pod-label == 'my-value' types: - Egress egress: - action: Allow destination: selector: color == 'red' ``` ### Tuning performance and latency[​](#tuning-performance-and-latency) Calico Cloud supports different DNS policy modes with different peformance and latency implications. The default mode for the iptables and nftables data planes is `DelayDeniedPacket`. The eBPF data plane defaults to `Inline` on kernels 5.17 (or RedHat 5.14) and newer and to `NoDelay` on older kernels. The Windows data plane support only `NoDelay` mode. With `NoDelay` mode, new pods sometimes fail to connect to domains that are allowed by a DNS policy. In these cases, the pod tries to connect to a domain before Calico Cloud has finished processing the DNS policy that allows the connection. As soon as the processing is complete, the pod is able to connect. For more information, see [DNSPolicyMode](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig#dnspolicymode) and [BPFDNSPolicyMode](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig#bpfdnspolicymode). ## Additional resources[​](#additional-resources) To change the default DNS trusted servers, use the [DNSTrustedServers parameter](https://docs.tigera.io/calico-cloud/reference/component-resources/node/felix/configuration). For more detail about the relevant resources, see [GlobalNetworkSet](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkset), [GlobalNetworkPolicy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy), [NetworkPolicy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) and [NetworkSet](https://docs.tigera.io/calico-cloud/reference/resources/networkset). ### Application layer policies to control ingress traffic ## [📄️ Enable and enforce application layer policies](https://docs.tigera.io/calico-cloud/network-policy/application-layer-policies/alp) [Enforce application layer policies in your cluster to configure access controls based on L7 attributes.](https://docs.tigera.io/calico-cloud/network-policy/application-layer-policies/alp) ## [📄️ Application layer policy tutorial](https://docs.tigera.io/calico-cloud/network-policy/application-layer-policies/alp-tutorial) [Learn how to apply ALP to your workloads and control ingress traffic.](https://docs.tigera.io/calico-cloud/network-policy/application-layer-policies/alp-tutorial) ### Enable and enforce application layer policies Application layer policies let you configure access controls based on L7 attributes. ## Before you begin[​](#before-you-begin) ### Limitations[​](#limitations) - Application layer policy is not compatible with a service mesh such as Istio. - Application layer policy can restrict only *ingress* traffic. It can not restrict egress traffic. - Support for L7 attributes is limited to HTTP method and URL exact/prefix path. - Supported protocols are limited to TCP-based protocols (for example, HTTP, HTTPS, or gRPC). - Application layer policies apply to the entire cluster. They can not be namespaced. - Logs for application layer polices are not included with other L7 logs in Service Graph. To view logs for application layer policies, you must [view them in Kibana](https://docs.tigera.io/calico-cloud/observability/elastic/l7/configure#kibana). - Application layer policy is supported only on Kubernetes 1.29 and later. > **INFO:** When you enable application layer policy for a deployment, all of that deployment's pods will restart Active connections may be disrupted. ### Enable application layer policy (ALP)[​](#enable-application-layer-policy-alp) Create an (or edit an existing ApplicationLayer) resource with the `sidecarInjection` field to `Enabled`: ```bash kubectl apply -f - < -n -p '{"spec":{"template":{"metadata":{"labels":{"applicationlayer.projectcalico.org/sidecar":"true"},"annotations":{"applicationlayer.projectcalico.org/policy":"Enabled"}}}}}' ``` ### Enforce application layer policies for ingress traffic[​](#enforce-application-layer-policies-for-ingress-traffic) You can restrict ingress traffic using HTTP match criteria using Global network policy. For a list of all HTTP match parameters, see [Global network policy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy). In the following example, the trading app is allowed ingress traffic only for HTTP GET requests that match the exact path /projects/calico, or that begins with the prefix, /users. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: customer spec: selector: app == 'tradingapp' ingress: - action: Allow http: methods: ["GET"] paths: - exact: "/projects/calico" - prefix: "/users" egress: - action: Allow ``` ### Disable application layer policies[​](#disable-application-layer-policies) To disable the policies, do one of the following steps: - Recreate your application pods without the sidecar label and annotations. - Remove the `applicationLayerPolicy` field entirely. - Remove the `sidecarInjection` field entirely, if sidecar mode was being used only for ALP. - Delete the ApplicationLayer\` custom resource if only ALP was enabled on it. ### Application layer policy tutorial This tutorial shows how to use Calico Cloud application layer policy to restrict ingress traffic for applications and microservices. ### Install the demo application[​](#install-the-demo-application) We will use a simple microservice application to demonstrate Calico Cloud application layer policy. The [YAO Bank](https://github.com/projectcalico/yaobank) application creates a customer-facing web application, a microservice that serves up account summaries, and an [etcd](https://github.com/coreos/etcd) datastore. ```bash kubectl apply -f https://docs.tigera.io/files/10-yaobank.yaml ``` ```bash namespace/yaobank configured service/database created serviceaccount/database created deployment.apps/database created service/summary created serviceaccount/summary created deployment.apps/summary created service/customer created serviceaccount/customer created deployment.apps/customer created ``` > **SECONDARY:** You can also [view the manifest in your browser](https://docs.tigera.io/assets/files/10-yaobank-e41239c6a0de293afb2f1b3c6a44eee6.yaml). Verify that the application pods have been created and are ready. ```bash kubectl rollout status deploy/summary deploy/customer deploy/database ``` When the demo application is displayed, you will see three pods. ```text NAME READY STATUS RESTARTS AGE customer-2809159614-qqfnx 3/3 Running 0 21h database-1601951801-m4w70 3/3 Running 0 21h summary-2817688950-g1b3n 3/3 Running 0 21h ``` ## Set up[​](#set-up) - A Calico Cloud cluster is running with application layer policy enabled - Cluster has three microservices: customer, database, summary - The customer web service should not have access to the backend database, but should have access to clients outside the cluster Imagine what would happen if an attacker were to gain control of the customer web pod in our application. Let's simulate this by executing a remote shell inside that pod. ```bash kubectl exec -ti customer- -c customer -- bash ``` Notice that from here, we get direct access to the backend database. For example, we can list all the entries in the database like this: ```bash curl http://database:2379/v2/keys?recursive=true | python -m json.tool ``` (Piping to `python -m json.tool` nicely formats the output.) ## Apply application layer policy[​](#apply-application-layer-policy) In this step, we get the application layer policy YAML and apply it. Note that the policy scope is cluster-wide. With a Calico Cloud policy, you can mitigate risks to the banking application. ```bash wget https://docs.tigera.io/files/30-policy.yaml kubectl create -f 30-policy.yaml ``` Let's examine this policy piece by piece. First, notice that an application layer policy looks like a regular Calico Cloud global network policy. The only difference you'll see is the ability to use the application layer policy parameters in global network policy. Another important difference is that you'll see HTTP traffic flows in the web console in features like Service Graph. Next, there are three policy objects, one for each microservice. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: customer spec: selector: app == 'customer' ingress: - action: Allow http: methods: ['GET'] egress: - action: Allow ``` The first policy protects the customer web app. Because this application is customer-facing, we do not restrict what can communicate with it. We do, however, restrict its communications to HTTP `GET` requests. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: summary spec: selector: app == 'summary' ingress: - action: Allow source: serviceAccounts: names: ['customer'] egress: - action: Allow ``` The second policy protects the account summary microservice. We know the only consumer of this service is the customer web app, so we restrict the source of incoming connections to the service account for the customer web app. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: database spec: selector: app == 'database' ingress: - action: Allow source: serviceAccounts: names: ["summary"] egress: - action: Allow ``` The third policy protects the database. Only the summary microservice should have direct access to the database. ### Verify the policy is working[​](#verify-the-policy-is-working) Let's verify our policy is working as intended. First, return to your browser and refresh to ensure policy enforcement has not broken the application. Next, return to the customer web app. Recall that we simulated an attacker gaining control of that pod by executing a remote shell inside it. ```bash kubectl exec -ti customer- -c customer bash ``` Repeat our attempt to access the database. ```bash curl -I http://database:2379/v2/keys?recursive=true ``` We omitted the JSON formatting because we do not expect to get a valid JSON response. This time we should get a `403 Forbidden` response. Only the account summary microservice has database access according to our policy. ### Policy for firewalls ## [🗃️ Fortinet firewall integrations](https://docs.tigera.io/calico-cloud/network-policy/policy-firewalls/fortinet-integration/) [3 items](https://docs.tigera.io/calico-cloud/network-policy/policy-firewalls/fortinet-integration/) ### Fortinet firewall integrations > **WARNING:** The Fortinet integration is deprecated and will be removed in a future release. If you want to use Calico Cloud with Fortinet or any other firewall, we recommend instead using an [egress gateway](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-on-prem). ## [📄️ Determine the best Calico Cloud/Fortinet solution](https://docs.tigera.io/calico-cloud/network-policy/policy-firewalls/fortinet-integration/overview) [Learn how to integrate Kubernetes clusters with existing Fortinet firewall workflows using Calico Cloud.](https://docs.tigera.io/calico-cloud/network-policy/policy-firewalls/fortinet-integration/overview) ## [📄️ Extend Kubernetes to Fortinet firewall devices](https://docs.tigera.io/calico-cloud/network-policy/policy-firewalls/fortinet-integration/firewall-integration) [Enable FortiGate firewalls to control traffic from Kubernetes workloads.](https://docs.tigera.io/calico-cloud/network-policy/policy-firewalls/fortinet-integration/firewall-integration) ## [📄️ Extend FortiManager firewall policies to Kubernetes](https://docs.tigera.io/calico-cloud/network-policy/policy-firewalls/fortinet-integration/fortimgr-integration) [Extend FortiManager firewall policies to Kubernetes with Calico Cloud](https://docs.tigera.io/calico-cloud/network-policy/policy-firewalls/fortinet-integration/fortimgr-integration) ### Determine the best Calico Cloud/Fortinet solution ## Big picture[​](#big-picture) Determine the best Calico Cloud/Fortinet solution to integrate Kubernetes clusters with your existing Fortinet firewall workflows. ## Value[​](#value) Many security teams must work within the confines of their existing IT security architecture, even though perimeter firewalls do not meet the needs of Kubernetes clusters. The Calico Cloud/Fortinet integration allows firewall administrators to leverage existing Fortinet security tools and workflows, continue meeting compliance requirements, while adopting Kubernetes orchestration using Calico Cloud at their own pace. ### Concepts[​](#concepts) The Calico Cloud/Fortinet integration provides the following solutions. You can you use them separately, or together without contention. ### Solution 1: Extend Kubernetes to Fortinet firewall devices[​](#solution-1-extend-kubernetes-to-fortinet-firewall-devices) **Use case**: Control egress traffic for Kubernetes clusters. **Problem**: Perimeter firewalls do not have the necessary information to act on traffic that leaves the cluster for Kubernetes workloads. **Solution**: The Calico Cloud/Fortinet integration leverages the power of Calico Cloud policy selectors to provide Kubernetes workload information to FortiManager and FortiGate devices. You create perimeter firewall policies in FortiManager and FortiGate that reference Kubernetes workloads. Policies are applied and enforced by FortiGate devices. And Firewall administrators can write cluster egress policies that reference Kubernetes workloads directly in Fortinet devices. ### Solution 2: Extend FortiManager firewall policies to Kubernetes[​](#solution-2-extend-fortimanager-firewall-policies-to-kubernetes) **Use case**: Control Kubernetes clusters directly and apply policy. **Problem**: To avoid disruption, teams need to leverage existing FortiManager as the primary user interface. **Solution**: Use FortiManager to create firewall policies that are applied as Calico Cloud network policies on Kubernetes workloads. Use the power of a Calico Cloud “higher-order tier” so Kubernetes policy is evaluated early in the policy processing order, but update policy using FortiManager UI. Use the Calico Cloud web console as a secondary interface to verify the integration and troubleshoot using logs. ## Next steps[​](#next-steps) - [Extend Kubernetes to Fortinet firewall devices](https://docs.tigera.io/calico-cloud/network-policy/policy-firewalls/fortinet-integration/firewall-integration) - [Extend FortiManager firewall policies to Kubernetes](https://docs.tigera.io/calico-cloud/network-policy/policy-firewalls/fortinet-integration/fortimgr-integration) ### Extend Kubernetes to Fortinet firewall devices > **WARNING:** The Fortinet integration is deprecated and will be removed in a future release. If you want to use Calico Cloud with Fortinet or any other firewall, we recommend instead using an [egress gateway](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-on-prem). ## Big picture[​](#big-picture) Use Calico Cloud network policy to control traffic from Kubernetes clusters in your FortiGate firewalls. ## Value[​](#value) As platform and security engineers, you want your apps to securely communicate with the external world. But you also want to secure the network traffic from the Kubernetes clusters using your Fortigate firewalls. Using the Fortinet/Calico Cloud integration, security teams can retain firewall responsibility, secure traffic using Calico Cloud network policy, which frees up time for ITOps. ## Concepts[​](#concepts) ## Integration at a glance[​](#integration-at-a-glance) This Calico Cloud/Fortinet integration workflow lets you control egress traffic leaving the Kubernetes cluster. You create perimeter firewall policies in FortiManager and FortiGate that reference Kubernetes workloads. Calico Cloud acts as a conduit, using the `tigera-firewall-controller` and global network policies to pass Kubernetes workload information to FortiManager and Fortigate devices where policies are applied and enforced. The basic workflow is: 1. Determine the Kubernetes pods that are allowed access outside the perimeter firewall. 2. Create Calico Cloud global network policies with selectors that match those pods. Each global network policy maps to an address group in the FortiGate firewall. 3. Deploy the `tigera firewall controller` in the Kubernetes cluster. 4. Create a ConfigMap with Fortinet firewall information. The `tigera firewall controller` reads the ConfigMap, gets the FortiGate firewall IP address, API token, and source IP address selection with `node` or `pod`. In your Kubernetes cluster, the controller populates pod IPs or Kubernetes node IPs of selector matching pods in Fortigate address group objects. ## Before you begin[​](#before-you-begin) **Supported versions** - FortiGate v6.2 - FortiManager v6.4 **Required** - IPv4 CIDR’s or IP addresses of all Kubernetes nodes; this is required for FortiManager to treat Kubernetes nodes as trusted hosts. **Recommended** - Experience creating and administering FortiGate/FortiManager firewall policies - Experience using [Calico Cloud tiers](https://docs.tigera.io/calico-cloud/reference/resources/tier) and [Global network policy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) ## How to[​](#how-to) - [Create tier and global network policy](#create-tier-and-global-network-policy) - [Configure FortiGate firewall to communicate with firewall controller](#configure-fortigate-firewall-to-communicate-with-firewall-controller) - [Configure FortiManager to communicate with firewall controller](#configure-fortimanager-to-communicate-with-firewall-controller) - [Create a config map for address selection in firewall controller](#create-a-config-map-for-address-selection-in-firewall-controller) - [Create a config map with FortiGate and FortiManager information](#create-a-config-map-with-fortigate-and-fortimanager-information) - [Install FortiGate ApiKey and FortiManager password as secrets](#install-fortigate-apikey-and-fortimanager-password-as-secrets) - [Deploy firewall controller in the Kubernetes cluster](#deploy-firewall-controller-in-the-kubernetes-cluster) ### Create tier and global network policy[​](#create-tier-and-global-network-policy) 1. Create a tier for organizing global network policies. Create a new [Tier](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/tiered-policy) to organize all Fortigate firewall global network policies in a single location. 2. Note the tier name to use in a later step for the FortiGate firewall information config map. 3. Create a GlobalNetworkPolicy for address group mappings. For example, a GlobalNetworkPolicy can select a set of pods that require egress access to external workloads. In the following GlobalNetworkPolicy, the firewall controller creates an address group named, `default.production-microservice1` in the Fortigate firewall. The members of `default.production-microservice1` address group include IP addresses of nodes. Each node can host one or more pods whose label selector match with `env == 'prod' && role == 'microservice1'`. Each GlobalNetworkPolicy maps to an address group in FortiGate firewall. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: default.production-microservice1 spec: selector: "env == 'prod' && role == 'microservice1'" types: - Egress egress: - action: Allow ``` ### Configure FortiGate firewall to communicate with firewall controller[​](#configure-fortigate-firewall-to-communicate-with-firewall-controller) 1. Determine and note the CIDR's or IP addresses of all Kubernetes nodes that can run the `tigera-firewall-controller`. Required to explicitly allow the `tigera-firewall-controller` to access the FortiGate API. 2. Create an Admin profile with read-write access to Address and Address Group Objects. For example: `tigera_api_user_profile` 3. Create a REST API Administrator, associate this user with the `tigera_api_user_profile` profile, and add the CIDR or IP address of your Kubernetes cluster nodes as trusted hosts. For example: `calico_enterprise_api_user` 4. Note the API key. ### Configure FortiManager to communicate with firewall controller[​](#configure-fortimanager-to-communicate-with-firewall-controller) 1. Determine and note the CIDR's or IP addresses of all Kubernetes nodes that can run the `tigera-firewall-controller`. Required to explicitly allow the tigera-firewall-controller to access the FortiManager API. 2. From system settings, create an Admin profile with Read-Write access for `Policy & Objects`. For example: `tigera_api_user_profile` 3. Create a JSON API administrator, associate this user with the `tigera_api_user_profile` profile, and add the CIDR or IP address of your Kubernetes cluster nodes as `Trusted Hosts`. 4. Note the username and password. ### Create a config map for address selection in firewall controller[​](#create-a-config-map-for-address-selection-in-firewall-controller) 1. Create a namespace for tigera-firewall-controller. ```bash kubectl create namespace tigera-firewall-controller ``` 2. Create a config map with FortiGate firewall information. For example: ```bash kubectl -n tigera-firewall-controller create configmap tigera-firewall-controller \ --from-literal=tigera.firewall.policy.selector="projectcalico.org/tier == 'default'" \ --from-literal=tigera.firewall.addressSelection="node" ``` **ConfigMap values** | Field | Enter values... | | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | tigera.firewall.policy.selector | The tier name with the global network policies with the Fortigate address group mappings. For example, this selects the global network policies in the `default` tier: \`tigera.firewall.policy.selector: "projectcalico.org/tier == 'default'" | | tigera.firewall.addressSelection | The addressSelection for outbound traffic leaving the cluster. For example, if outgoingNat is enabled in cluster and compute Node IP address is used "tigera.firewall.addressSelection == `node` or If pod IP address used then "tigera.firewall.addressSelection == `pod`" | ### Create a config map with FortiGate and FortiManager information[​](#create-a-config-map-with-fortigate-and-fortimanager-information) 1. In the [Fortigate ConfigMap manifest](https://downloads.tigera.io/ee/v3.22.0-2.0/manifests/fortinet-device-configmap.yaml), add your FortiGate firewall information in the data section, `tigera.firewall.fortigate`. Where: | Field | Description | | ------------------------ | --------------------------------------------------------------------------- | | name | FortiGate device name | | ip | FortiGate Management Ip address | | apikey | Secret in tigera-firewall-controller namespace, to store FortiGate's APIKey | | apikey.secretKeyRef.name | Name of the secret to store APIKey. | | apikey.secretKeyRef.key | Key name in the secret, which stores APIKey | For example: ```yaml - name: prod-eastcoast-1 ip: 1.2.3.1 apikey: secretKeyRef: name: fortigate-east1 key: apikey-fortigate-east1 - name: prod-eastcoast-2 ip: 1.2.3.2 apikey: secretKeyRef: name: fortigate-east2 key: apikey-fortigate-east2 ``` 2. In the [FortiManager ConfigMap manifest](https://downloads.tigera.io/ee/v3.22.0-2.0/manifests/fortinet-device-configmap.yaml), add your FortiManager information in the data section, `tigera.firewall.fortimgr`. Where: | Field | Description | | -------------------------- | ------------------------------------------------------------------------------ | | name | FortiManager device name | | ip | FortiManager Management Ip address | | adom | FortiManager ADOM name to manage kubernetes cluster. | | username | JSON api access account name to Read/Write FortiManager address objects. | | password | Secret in tigera-firewall-controller namespace, to store FortiManager password | | password.secretKeyRef.name | Name of the secret to store password. | | password.secretKeyRef.key | Key name in the secret, which stores password. | For example: ```yaml - name: prod-east1 ip: 1.2.4.1 username: api_user adom: root password: secretKeyRef: name: fortimgr-east1 key: pwd-fortimgr-east1 ``` > **SECONDARY:** If you are not using FortiManager in the integration, include only the following field in the ConfigMap data section. `tigera.firewall.fortimgr: |` 1. Apply the manifest. ```text kubectl apply -f https://downloads.tigera.io/ee/v3.22.0-2.0/manifests/fortinet-device-configmap.yaml ``` ### Install FortiGate ApiKey and FortiManager password as secrets[​](#install-fortigate-apikey-and-fortimanager-password-as-secrets) 1. Store each FortiGate API key as a secret in the `tigera-firewall-controller` namespace. For example, the FortiGate device, `prod-east1`, store its ApiKey as a secret name as `fortigate-east1`, with key as `apikey-fortigate-east1`. ```text kubectl create secret generic fortigate-east1 \ -n tigera-firewall-controller \ --from-literal=apikey-fortigate-east1= ``` 2. Store each FortiManager password as secret in the `tigera-firewall-controller` namespace. For example, for FortiMgr `prod-east1`, store its password as a secret name as `fortimgr-east1`, with key as `pwd-fortimgr-east1`. ```text kubectl create secret generic fortimgr-east1 \ -n tigera-firewall-controller \ --from-literal=pwd-fortimgr-east1= ``` ### Deploy firewall controller in the Kubernetes cluster[​](#deploy-firewall-controller-in-the-kubernetes-cluster) 1. Install your pull secret. ```text kubectl create secret generic tigera-pull-secret \ --from-file=.dockerconfigjson= \ --type=kubernetes.io/dockerconfigjson -n tigera-firewall-controller ``` 2. Apply the manifest. ```text kubectl apply -f https://downloads.tigera.io/ee/v3.22.0-2.0/manifests/fortinet.yaml ``` ## Verify the integration[​](#verify-the-integration) 1. Log in to the FortiGate firewall user interface. 2. Under **Policy & Objects**, click **Addresses**. 3. Verify that your Kubernetes-related address objects and address group objects are created with the following comments "Managed by Tigera Calico Cloud". Fof all FortiManagers that are configured to work with firewall-controller, log in to each FortiManager UI with the correct ADOM. 1. Click **Policy & Objects**, **Object Configuration**, \*\*Addresses. 2. Verify that your Kubernetes-related address objects and address group objects are created with the following comments "Managed by Tigera Calico Cloud". ## Additional resources[​](#additional-resources) - [Extend FortiManager firewall policies to Kubernetes](https://docs.tigera.io/calico-cloud/network-policy/policy-firewalls/fortinet-integration/fortimgr-integration) ### Extend FortiManager firewall policies to Kubernetes > **WARNING:** The Fortinet integration is deprecated and will be removed in a future release. If you want to use Calico Cloud with Fortinet or any other firewall, we recommend instead using an [egress gateway](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-on-prem). ## Big picture[​](#big-picture) Use FortiManager firewall policies to secure workloads in your Kubernetes cluster. ## Value[​](#value) The Calico Cloud/Fortinet integration lets you control Kubernetes clusters directly and apply policy using the FortiManager UI as the primary interface. This allows firewall administrators to leverage existing tools and workflows as they learn and adopt Kubernetes orchestration at their own pace. ## Concepts[​](#concepts) ### Integration at a glance[​](#integration-at-a-glance) This Calico Cloud/Fortinet solution lets you directly control Kubernetes policies using FortiManager. The basic workflow is: 1. Determine the Kubernetes pods that you want to securely communicate with each other. 2. Label these pods using a key-value pair where key is the `tigera.io/address-group`, and value is the pod matching a label name. 3. In the FortiManager, select the cluster’s ADOM, and create an address group using the key-value pair associated with the pods. 4. Create firewall policies using the address groups for IPv4 Source address and IPv4 Destination Address, and select services and actions as you normally would to allow or deny the traffic. Under the covers, the Calico Cloud integration controller periodically reads the FortiManager firewall policies for your Kubernetes cluster, converts them to Calico Cloud global network policies, and applies them to clusters. 5. Use the Calico Cloud web console to verify the integration, and then FortiManager UI to make all updates to policy rules. > **SECONDARY:** The default value for reading FortiManager firewall policies is three seconds. To change the value, modify environment variable FW\_FORTIMGR\_EW\_POLL\_INTERVAL in FortiManager integration manifest; units are in seconds. ## Before you begin[​](#before-you-begin) **Supported version** - FortiManager v6.4 **Required** - IPv4 CIDR’s or IP addresses of all Kubernetes nodes; this is required for FortiManager to treat Kubernetes nodes as trusted hosts. **Recommended** - Experience with [tiered policy](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/tiered-policy) and [global network policy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) - Experience creating and administering FortiGate/FortiManager firewall policies ## How to[​](#how-to) - [Create a tier](#create-a-tier) - [Configure FortiManager to communicate with firewall controller](#configure-fortimanager-to-communicate-with-firewall-controller) - [Create a FortiManager config map](#create-a-fortimanager-config-map) - [Install FortiManager password as secrets](#install-fortimanager-password-as-secrets) - [Deploy the firewall controller in the Kubernetes cluster](#deploy-the-firewall-controller-in-the-kubernetes-cluster) - [Verify the integration](#verify-the-integration) ### Create a tier[​](#create-a-tier) Create a [Calico Cloud tier](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/tiered-policy) in the Calico Cloud web console for each Kubernetes cluster you want to secure. We recommend that you create a new tier (rather than reusing an existing tier) for all global network policies created by the Calico Cloud integration controller. ## Configure FortiManager to communicate with firewall controller[​](#configure-fortimanager-to-communicate-with-firewall-controller) 1. Determine and note the CIDR’s or IP addresses of all Kubernetes nodes that can run the `tigera-firewall-controller`. This is required to explicitly allow the `tigera-firewall-controller` to access the FortiManager API. 2. From system settings, create an Admin profile with Read-Write access for `Policy & Objects`. For example: `tigera_api_user_profile` 3. Create a JSON API administrator and associate this user with the `tigera_api_user_profile` profile and add CIDR or IP address of your Kubernetes cluster nodes as `trusted hosts`. 4. Note the username and password. ## Create a FortiManager config map[​](#create-a-fortimanager-config-map) 1. Create a namespace for the tigera-firewall-controller. ```bash kubectl create namespace tigera-firewall-controller ``` 2. In this [FortiManager ConfigMap manifest](https://downloads.tigera.io/ee/v3.22.0-2.0/manifests/fortimanager-device-configmap.yaml), add your FortiManager device information in the data section: `tigera.firewall.fortimanager-policies`. For example: ```yaml tigera.firewall.fortimanager-policies: | - name: prod-east1 ip: 3.2.1.4 username: api_user adom: root tier: packagename: sacramento password: secretKeyRef: name: fortimgr-east1 key: pwd-fortimgr-east1 ``` Where: | Field | Description | | -------------------------- | ---------------------------------------------------------------------------------------------------------------- | | name | FortiManager device name. | | ip | FortiManager Management IP address. | | adom | FortiManager ADOM name that manages Kubernetes cluster. | | packagename | FortiManager Firewall package. All firewall rules targeted for Kubernetes cluster are packed under this package. | | username | JSON api access account name to Read/Write FortiManager address objects. | | password | Secret in tigera-firewall-controller namespace, to store FortiManager password | | tier | Tier name you created in the Calico Cloud web console | | password.secretKeyRef.name | Name of the secret to store password. | | password.secretKeyRef.key | Key name in the secret, which stores password. | 3. Apply the manifest. ```bash kubectl apply -f https://downloads.tigera.io/ee/v3.22.0-2.0/manifests/fortimanager-device-configmap.yaml ``` ## Install FortiManager password as secrets[​](#install-fortimanager-password-as-secrets) Store each FortiManager password as a secret in the `tigera-firewall-controller` namespace. For example, in the ConfigMap for FortiMgr `prod-east1`, store its password as a secret name as `fortimgr-east1`, with key as `pwd-fortimgr-east1`. ```bash kubectl create secret generic fortimgr-east1 \ -n tigera-firewall-controller \ --from-literal=pwd-fortimgr-east1= ``` ### Deploy the firewall controller in the Kubernetes cluster[​](#deploy-the-firewall-controller-in-the-kubernetes-cluster) 1. Install your pull secret. ```bash kubectl create secret generic tigera-pull-secret \ --from-file=.dockerconfigjson= \ --type=kubernetes.io/dockerconfigjson -n tigera-firewall-controller ``` 2. Apply the manifest. ```bash kubectl apply -f https://downloads.tigera.io/ee/v3.22.0-2.0/manifests/fortimanager.yaml ``` ## Verify the integration[​](#verify-the-integration) 1. Log in to FortiManager with the correct ADOM. 2. Select **Policy & Objects**, **Object Configuration**, and create new **Address Groups**. 3. Click **Policy packages** and select the Package assigned to your Kubernetes cluster. 4. Create a test firewall policy with the following fields: Name, IPv4 Source Address, IPv4 Destination Address, Service and Action. 5. Log in to the Calico Cloud web console, and under the tier that you specified in the ConfigMap, verify that the GlobalNetworkPolicies are created. ### Protect Kubernetes nodes ## Big picture[​](#big-picture) Secure Kubernetes nodes with host endpoints managed by Calico Cloud. ## Value[​](#value) Calico Cloud can automatically create host endpoints for your Kubernetes nodes. This means Calico Cloud can manage the lifecycle of host endpoints as your cluster evolves, ensuring nodes are always protected by policy. ## Concepts[​](#concepts) ## Host endpoints[​](#host-endpoints) Each host has one or more network interfaces that it uses to communicate externally. You can represent these interfaces in Calico Cloud using host endpoints and then use network policy to secure them. Calico Cloud host endpoints can have labels, and they work the same as labels on workload endpoints. The network policy rules can apply to both workload and host endpoints using label selectors. Automatic host endpoints secure all of the host's interfaces (i.e. in Linux, all the interfaces in the host network namespace). They are created by setting `interfaceName: "*"`. ## Automatic host endpoints[​](#automatic-host-endpoints) Calico Cloud creates a wildcard host endpoint for each node, with the host endpoint containing the same labels and IP addresses as its corresponding node. Calico Cloud will ensure these managed host endpoints maintain the same labels and IP addresses as its node by periodic syncs. This means that policy targeting these automatic host endpoints will function correctly with the policy put in place to select those nodes, even if over time the node's IPs or labels change. Calico Cloud is also able to create and manage custom host endpoints for nodes, based on user specified templates. This allows you to fine-tune which interfaces should be included in the host endpoint and for which nodes these host endpoints should be created. The host endpoint templates can be enabled by updating [KubeControllersConfig](https://docs.tigera.io/calico-cloud/reference/resources/kubecontrollersconfig). Automatic host endpoints are differentiated from other host endpoints by the label `projectcalico.org/created-by: calico-kube-controllers`. Enable or disable automatic host endpoints by configuring the default KubeControllersConfiguration resource. ## Before you begin[​](#before-you-begin) **Unsupported** - GKE ## How to[​](#how-to) - [Enable automatic host endpoints](#enable-automatic-host-endpoints) - [Apply network policy to automatic host endpoints](#apply-network-policy-to-automatic-host-endpoints) ### Enable automatic host endpoints[​](#enable-automatic-host-endpoints) To enable automatic host endpoints, edit the default KubeControllersConfiguration instance, and set `spec.controllers.node.hostEndpoint.autoCreate` to `true`: ```bash kubectl patch kubecontrollersconfiguration default --patch='{"spec": {"controllers": {"node": {"hostEndpoint": {"autoCreate": "Enabled"}}}}}' ``` If successful, host endpoints are created for each of your cluster's nodes: ```bash kubectl get heps -o wide ``` The output may look similar to this: ```text kubectl get heps -o wide NAME CREATED AT ip-172-16-101-147.us-west-2.compute.internal-auto-hep 2021-05-12T22:16:47Z ip-172-16-101-54.us-west-2.compute.internal-auto-hep 2021-05-12T22:16:47Z ip-172-16-101-79.us-west-2.compute.internal-auto-hep 2021-05-12T22:16:47Z ip-172-16-101-9.us-west-2.compute.internal-auto-hep 2021-05-12T22:16:47Z ip-172-16-102-63.us-west-2.compute.internal-auto-hep 2021-05-12T22:16:47Z ``` ### Use custom templates for host endpoints[​](#use-custom-templates-for-host-endpoints) Template HostEndpoints allow for tailored, automatic HostEndpoint generation. Defined in spec.controllers.node.hostEndpoint.templates, they enable creation based on specific Node or interface criteria, offering greater customization compared to default automatic HostEndpoints. To enable template host endpoints and disable default host endpoints, edit the default KubeControllersConfiguration instance, and update `spec.controllers.node.hostEndpoint.templates` with your custom templates spec. 1. Create a custom templates spec: ```bash cat << 'EOF' > hostEndpointTemplate.yaml spec: controllers: node: hostEndpoint: autoCreate: Enabled createDefaultHostEndpoint: Disabled templates: - generateName: custom-host-endpoint interfaceCIDRs: - 192.168.0.0/16 nodeSelector: "!has(node-role.kubernetes.io/control-plane)" labels: custom-label: template-host-endpoint EOF ``` 2. Merge the templates spec with the `kubecontrollersconfiguration` resource: ```bash kubectl patch kubecontrollersconfiguration default --patch-file hostEndpointTemplate.yaml --type merge ``` 3. Verify that your `kubecontrollersconfiguration` resource is updated: ```bash kubectl get kubecontrollersconfiguration default -o yaml ``` The output should look similar to this: ```text apiVersion: projectcalico.org/v3 kind: KubeControllersConfiguration metadata: name: default spec: controllers: node: hostEndpoint: autoCreate: Enabled createDefaultHostEndpoint: Enabled templates: - generateName: custom-host-endpoint interfaceCIDRs: - 192.168.0.0/16 labels: custom-label: template-host-endpoint ``` 4. If successful, host endpoints are created for nodes matching your node selector: ```bash kubectl get heps -o=custom-columns='NAME:.metadata.name,NODE:.spec.node,INTERFACES:.spec.interfaceName,IPS:.spec.expectedIPs,PROFILES:.spec.profiles' ``` The output may look similar to this: ```text NAME NODE INTERFACE IPS PROFILES 192.168.83.64.compute.internal-custom-host-endpoint-auto-hep 192.168.83.64.compute.internal 192.168.83.64 projectcalico-default-allow 192.168.0.128.compute.internal-custom-host-endpoint-auto-hep 192.168.0.128.compute.internal 192.168.0.128 projectcalico-default-allow 192.168.157.128.compute.internal-custom-host-endpoint-auto-hep 192.168.157.128.compute.internal 192.168.157.128 projectcalico-default-allow ``` ### Apply network policy to automatic host endpoints[​](#apply-network-policy-to-automatic-host-endpoints) To apply policy that targets all Kubernetes nodes, first add a label to the nodes. The label will be synced to their automatic host endpoints. For example, to add the label **kubernetes-host** to all nodes and their host endpoints: ```bash kubectl label nodes --all kubernetes-host= ``` And an example policy snippet: ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: all-nodes-policy spec: selector: has(kubernetes-host) # ``` To select a specific set of host endpoints (and their corresponding Kubernetes nodes), use a policy selector that selects a label unique to that set of host endpoints. For example, if we want to add the label **environment=dev** to nodes named node1 and node2: ```bash kubectl label node node1 environment=dev kubectl label node node2 environment=dev ``` With the labels in place and automatic host endpoints enabled, host endpoints for node1 and node2 will be updated with the **environment=dev** label. We can write policy to select that set of nodes with a combination of selectors: ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: some-nodes-policy spec: selector: has(kubernetes-host) && environment == 'dev' # ``` ## Tutorial[​](#tutorial) This tutorial will lock down Kubernetes node ingress to only allow SSH and required ports for Kubernetes to function. We will apply two policies: one for the control plane nodes. and one for the worker nodes. > **SECONDARY:** Note: This tutorial was tested on a cluster created with kubeadm v1.18.2 on AWS, using a "stacked etcd" [topology](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/ha-topology/). Stacked etcd topology means the etcd pods are running on the masters. kubeadm uses stacked etcd by default. If your Kubernetes cluster is on a different platform, is running a variant of Kubernetes, or is running a topology with an external etcd cluster, please review the required ports for control plane and worker nodes in your cluster and adjust the policies in this tutorial as needed. First, let's restrict ingress traffic to the control plane nodes. The ingress policy below contains three rules. The first rule allows access to the API server port from anywhere. The second rule allows all traffic to localhost, which allows Kubernetes to access control plane processes. These control plane processes includes the etcd server client API, the scheduler, and the controller-manager. This rule also allows localhost access to the kubelet API and calico/node health checks. And the final rule allows the etcd pods to peer with each other and allows the masters to access each others kubelet API. If you have not modified the failsafe ports, you should still have SSH access to the nodes after applying this policy. Now apply the ingress policy for the Kubernetes masters: ```text kubectl apply -f - << EOF apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: ingress-k8s-masters spec: selector: has(node-role.kubernetes.io/master) # This rule allows ingress to the Kubernetes API server. ingress: - action: Allow protocol: TCP destination: ports: # kube API server - 6443 # This rule allows all traffic to localhost. - action: Allow destination: nets: - 127.0.0.0/8 # This rule is required in multi-master clusters where etcd pods are colocated with the masters. # Allow the etcd pods on the masters to communicate with each other. 2380 is the etcd peer port. # This rule also allows the masters to access the kubelet API on other masters (including itself). - action: Allow protocol: TCP source: selector: has(node-role.kubernetes.io/master) destination: ports: - 2380 - 10250 EOF ``` Note that the above policy selects the standard **node-role.kubernetes.io/master** label that kubeadm sets on control plane nodes. Next, we need to apply policy to restrict ingress to the Kubernetes workers. Before adding the policy we will add a label to all of our worker nodes, which then gets added to its automatic host endpoint. For this tutorial we will use **kubernetes-worker**. An example command to add the label to worker nodes: ```bash kubectl get node -l '!node-role.kubernetes.io/master' -o custom-columns=NAME:.metadata.name | tail -n +2 | xargs -I{} kubectl label node {} kubernetes-worker= ``` The workers' ingress policy consists of two rules. The first rule allows all traffic to localhost. As with the masters, the worker nodes need to access their localhost kubelet API and calico/node healthcheck. The second rule allows the masters to access the workers kubelet API. Now apply the policy: ```text kubectl apply -f - << EOF apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: ingress-k8s-workers spec: selector: has(kubernetes-worker) # Allow all traffic to localhost. ingress: - action: Allow destination: nets: - 127.0.0.0/8 # Allow only the masters access to the nodes kubelet API. - action: Allow protocol: TCP source: selector: has(node-role.kubernetes.io/master) destination: ports: - 10250 EOF ``` ## Additional resources[​](#additional-resources) - [Apply policy to Kubernetes node ports](https://docs.tigera.io/calico-cloud/network-policy/beginners/services/kubernetes-node-ports) - [Global network policy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) - [Host endpoints](https://docs.tigera.io/calico-cloud/reference/resources/hostendpoint) ### Apply policy to forwarded traffic ## Big picture[​](#big-picture) Enforce Calico Cloud policy on traffic transiting a host that is used as a router or NAT gateway. ## Value[​](#value) If your host has multiple network interfaces, and is configured as a router, or NAT gateway between two different networks, you may want to enforce policy on traffic as it moves between the networks. In this configuration, often neither the source or destination is a Calico Cloud endpoint, so policy enforcement at the endpoint is not available. You can centrally manage the firewall policy on a fleet of such hosts using the same policy language as the rest of Calico Cloud. ## Concepts[​](#concepts) ### Workload endpoints and host endpoints[​](#workload-endpoints-and-host-endpoints) The following figure shows a host with two network interfaces: eth0 and eth1. We call these **host endpoints (HEPs)**. The host also runs two guest workloads (VMs or containers). We call the virtual interfaces to the guests, **workload endpoints (WEPs)**. Each has a corresponding configuration object on the Calico Cloud API called HostEndpoint and WorkloadEndpoint, respectively. The `HostEndpoint API` object is optional, and Calico Cloud does not enforce any policy on the HEP if the API object is missing. The `WorkloadEndpoint API` object is required, and is automatically managed by the cluster orchestrator plugin (for example, Kubernetes or OpenStack). Several connections are shown in the figure, numbered 1 through 4. For example, connection 1 ingresses over HEP eth0, is forwarded, and then ingresses Workload A’s WEP. Calico policies select which WEPs or HEPs they apply to. So, for example an ingress policy that selects Workload A’s WEP will apply to connections as shown in number 1. ![Host-forward-traffic](https://docs.tigera.io/assets/images/host-forward-traffic-e17a4eb534e02e1b84682193d8a478f0.png) ### applyOnForward[​](#applyonforward) By default, Calico Cloud global network policies set **applyOnForward to false**. When set to false on policies that select HEPs, the policies are applied only to traffic that originates or terminates on the host, for example: connection 4 (Node process). Connections 1-3 are unaffected by policies that select the HEP, but have applyOnForward set to false. In contrast, if applyOnForward is set to true for a policy that selects a HEP, that policy can apply to all connections 1-4. For example: - Ingress policy on HEP eth0 affects connections 1 and 2 - Egress policy on HEP eth1 affects connections 2, 3, and 4 There are also different default action semantics for **applyOnForward: true policy** versus **applyOnForward: false policy**. An applyOnForward: true policy affects all traffic through the HEP (connections 1-4). If no applyOnForward policy selects the HEP and direction (ingress versus egress), then forwarded traffic is allowed. If no policy (regardless of applyOnForward) selects the HEP and direction, then local traffic is denied. | **HEP defined?** | **Traffic Type** | **applyOnForward defined?** | **Any policy defined?** | **Default Action** | | ---------------- | ---------------- | --------------------------- | ----------------------- | ------------------ | | No | Any | n/a | n/a | Allow | | Yes | Forwarded | No | Any | Allow | | Yes | Forwarded | Yes | Yes | Deny | | Yes | Local | n/a | No | Deny | | Yes | Local | n/a | Yes | Deny | **Calico Cloud namespaced network policies** do not have an applyOnForward setting. HEPs are always cluster global, not namespaced, so network policies cannot select them. ### preDNAT policy[​](#prednat-policy) Hosts are often configured to perform Destination Network Address Translation before forwarding certain packets. A common example of this in cloud computing is when the host acts as a reverse-proxy to load balance service requests for a set of backend workload instances. To apply policy to a specific example of such a reverse-proxy, see [Kubernetes nodePorts](https://docs.tigera.io/calico-cloud/network-policy/beginners/services/kubernetes-node-ports). When preDNAT is set to false on a global network policy, the policy rules are evaluated on the connection after DNAT is performed. False is the default. When preDNAT is set to true, the policy rules are evaluated on the connection before DNAT has been performed. If you set preDNAT policy to true, you must set applyOnForward to true, and preDNAT policy must only include Ingress policies. ### Host endpoints with interfaceName: `*`[​](#host-endpoints-with-interfacename-) HostEndpoint API objects can be created with the name of the host interface (as reported by ip link or similar), or they can be created with interfaceName set to `*`, which means all host interfaces on the given node, including the interfaces between the host to any WEPs on that host. With interfaceName set to a particular interface, any policies that select the HEP apply only if the traffic goes through the named interface. With it set to `*`, policies that select the HEP apply regardless of the interface. This is particularly relevant when you want to enforce policy for a host that also runs guest workloads like VMs or Pods. Traffic from local workloads to reverse-proxy IPs or ports do not traverse any external interfaces, and thus a HEP with interfaceName set to \* is required in order for policy to apply to them. ## How to[​](#how-to) ### Control forwarded traffic in or out of particular networks[​](#control-forwarded-traffic-in-or-out-of-particular-networks) 1. Choose a labeling scheme for your Host Endpoints (network interfaces). For example, if you have an application network and management network, you might choose the labels **network = application** and **network = management**. 2. Write GlobalNetworkPolicies expressing your desired rules. - applyOnForward set to true. - Use the **selector:** to choose which Host Endpoints to apply policy. 3. Create the HostEndpoint objects on the `Calico Cloud API`. - Label the HostEndpoints according to the label scheme you developed in step 1. - We recommend that you create policies before you create the Host Endpoints. This ensures that all policies exist before Calico Cloud starts enforcing. ## Tutorial[​](#tutorial) Let’s say I have a host that has two network interfaces: - eth0 - connects to the main datacenter network for application traffic - eth1 - connects to a special maintenance network My goal is to allow SSH traffic to be forwarded to the maintenance network, but to drop all other traffic. I choose the following label scheme: - network = application for the main datacenter network for application traffic - network = maintenance for the maintenance network I create the GlobalNetworkPolicy that allows SSH traffic (default deny is implicit in this case). ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: allow-ssh-maintenance spec: selector: network == 'maintenance' applyOnForward: true types: - Ingress ingress: # Allow SSH - action: Allow protocol: TCP destination: ports: - 22 ``` Save this as allow-ssh-maintenance.yaml. Apply the policy to the cluster: ```bash kubectl create -f allow-ssh-maintenance.yaml ``` Finally, create the host endpoint for the interface that connects to the maintenance network. ```yaml apiVersion: projectcalico.org/v3 kind: HostEndpoint metadata: name: myhost.eth1 labels: network: maintenance spec: interfaceName: eth1 node: myhost expectedIPs: - 192.168.0.45 ``` Replace myhost with the node name Calico Cloud uses, and the expected IPs with the actual interface IP address(es). Save this file as hep.yaml. Apply the host endpoint to the cluster: ```bash kubectl create -f hep.yaml ``` For completeness, you could also create a HostEndpoint for eth0, but because we have not written any policies for the application network yet, you can omit this step. ## Viewing flow logs for host forwarded traffic[​](#viewing-flow-logs-for-host-forwarded-traffic) When using `applyOnForward` and `preDNAT` policies with host endpoints, you can monitor and troubleshoot policy enforcement by examining flow logs. Calico Cloud captures detailed information about how policies are applied to forwarded traffic. ### Configure policy information in flow logs[​](#configure-policy-information-in-flow-logs) To see policy information in flow logs, you must enable flow logs for forwarding decisions in the FelixConfiguration resource: ```yaml apiVersion: projectcalico.org/v3 kind: FelixConfiguration metadata: name: default spec: flowLogsPolicyScope: AllPolicies ``` This configuration enables tracking the flows transiting through the node interfaces within `policies` field in flow logs. The `policies` field contains detailed information about which policies acted on the traffic. ### Understanding the `transit_policies` field in flow logs[​](#understanding-the-transit_policies-field-in-flow-logs) The `policies` field in flow logs contains sub-fields that show different policy evaluation results: - **`transit_policies`**: All host endpoint policies with `applyOnForward: true` that applied an action to the flow while in transit through the node (for forwarded traffic between network interfaces). Note that policies with `preDNAT: true` will also appear here when they apply to ingress traffic before DNAT processing. Each policy entry in these fields follows this format: ```text ||.|| ``` ### Example flow log for applyOnForward policy[​](#example-flow-log-for-applyonforward-policy) Using the tutorial example from this document (SSH traffic to maintenance network), here's what a flow log entry might look like: ```json { "source_ip": "10.0.0.1", "source_name": "client-pods-gkh45", "source_name_aggr": "client-pods-*", "source_namespace": "client-pods-namespace", "dest_ip": "192.168.1.10", "dest_name": "maintenance-server-gkh45", "dest_name_aggr": "maintenance-server-*", "dest_namespace": "maintenance-server-namespace", "dest_port": 22, "dest_type": "wep", "proto": "tcp", "action": "allow", "reporter": "dst, fwd", "policies": { "all_policies": [ "0|__PROFILE__|__PROFILE__.kns.maintenance-server|allow|0" ], "enforced_policies": [ "0|__PROFILE__|__PROFILE__.kns.maintenance-server|allow|0" ], "pending_policies": [ "0|__PROFILE__|__PROFILE__.kns.maintenance-server|allow|0" ], "transit_policies": [ "0|default|default.allow-ssh-maintenance|allow|0" ] }, "bytes_in": 1234, "bytes_out": 5678, "packets_in": 10, "packets_out": 8, "host": "my-host" } ``` **Key points about this flow log:** - `reporter` is `dst, fwd`, indicating this flow was reported by both the destination workload endpoint and the forwarding node. The `enforced_policies` shows policies that applied to the destination workload's ingress traffic, while `transit_policies` shows the host endpoint policy that applied as the traffic was forwarded through the node. - The `policies.transit_policies` field shows that the `allow-ssh-maintenance` GlobalNetworkPolicy allowed the traffic - The rule index `0` indicates the first rule in the policy matched (the SSH allow rule) ### Example flow log for preDNAT policy[​](#example-flow-log-for-prednat-policy) Here's an example of a GlobalNetworkPolicy with `preDNAT: true` that allows HTTP traffic incoming to the cluster from external sources: ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: allow-external-http spec: selector: network = 'maintenance' applyOnForward: true preDNAT: true types: - Ingress ingress: # Allow HTTP from external sources to NodePort services - action: Allow protocol: TCP source: nets: - 34.0.113.0/24 # External network destination: ports: - 30080 # NodePort ``` For preDNAT policies, flow logs display the original destination IP and port before DNAT is performed. When traffic originates from outside the cluster (for example, an external client accesses a NodePort service), the flow log captures the pre-DNAT source and destination details. Here's an example: ```json { "source_ip": "34.0.113.100", "source_name": "-", "source_name_aggr": "pub", "source_namespace": "-", "source_type": "net", "dest_ip": "66.11.155.1", "dest_name": "-", "dest_name_aggr": "pub", "dest_namespace": "-", "dest_type": "net", "dest_port": 30080, "proto": "tcp", "action": "allow", "reporter": "fwd", "policies": { "transit_policies": [ "0|default|default.allow-external-http|allow|0" ] }, "bytes_in": 2048, "bytes_out": 4096, "packets_in": 15, "packets_out": 12, "host": "my-host" } ``` **Key points about this preDNAT flow log:** - `source_name_aggr` is `pub`, indicating external traffic from outside the cluster - `dest_name_aggr` is `pub`, indicating the NodePort's external facing IP - `dest_port` shows the original NodePort (30080) before DNAT translation to the service port - The `reporter` is `fwd`, indicating this policy acted on forwarded ingress traffic at the host endpoint - The `policies.transit_policies` field shows the preDNAT policy that allowed the traffic before DNAT processing ## Additional resources[​](#additional-resources) - [Host endpoint](https://docs.tigera.io/calico-cloud/reference/resources/hostendpoint) - [Workload endpoint](https://docs.tigera.io/calico-cloud/reference/resources/workloadendpoint) - [Global network policy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) ### Policy for extreme traffic ## [📄️ Enable extreme high-connection workloads](https://docs.tigera.io/calico-cloud/network-policy/extreme-traffic/high-connection-workloads) [Create a Calico network policy rule to bypass Linux conntrack for traffic to workloads that experience extremely large number of connections.](https://docs.tigera.io/calico-cloud/network-policy/extreme-traffic/high-connection-workloads) ## [📄️ Defend against DoS attacks](https://docs.tigera.io/calico-cloud/network-policy/extreme-traffic/defend-dos-attack) [Define DoS mitigation rules in Calico Cloud policy to quickly drop connections when under attack. Learn how rules use eBPF and XDP, including hardware offload when available.](https://docs.tigera.io/calico-cloud/network-policy/extreme-traffic/defend-dos-attack) ### Enable extreme high-connection workloads ## Big picture[​](#big-picture) Use a Calico Cloud network policy rule to bypass Linux conntrack for traffic to workloads that experience extremely large number of connections. ## Value[​](#value) When the number of connections on a node exceeds the number of connections that Linux conntrack can track, connections can be rejected or dropped. Calico Cloud network policy can be used to selectively bypass Linux conntrack for traffic to/from these types of workloads. ## Concepts[​](#concepts) ### Linux conntrack[​](#linux-conntrack) Connection tracking (“conntrack”) is a core feature of the Linux kernel’s networking stack. It allows the kernel to keep track of all logical network connections or flows, and thereby identify all of the packets that make up each flow so they can be handled consistently together. Conntrack is an essential part of the mainline Linux network processing pipeline, normally improving performance, and enabling NAT and stateful access control. ### Extreme high-connection workloads[​](#extreme-high-connection-workloads) Some niche workloads handling extremely high number of simultaneous connections, or very high rate of short lived connections, can exceed the maximum number of connections Linux conntrack is able to track. One real world example of such a workload is an extreme scale memcached server handling 50k+ connections per second. ### Calico Cloud doNotTrack network policy[​](#calico-cloud-donottrack-network-policy) The Calico Cloud global network policy option, **doNotTrack**, indicates to apply the rules in the policy before connection tracking, and that packets allowed by these rules should not be tracked. The policy is applied early in the Linux packet processing pipeline, before any regular network policy rules, and independent of the policy order field. Unlike normal network policy rules, doNotTrack network policy rules are stateless, meaning you must explicitly specify rules to allow return traffic that would normally be automatically allowed by conntrack. For example, for a server on port 999, the policy must include an ingress rule allowing inbound traffic to port 999, and an egress rule to allow outbound traffic from port 999. In a doNotTrack policy: - Ingress rules apply to all incoming traffic through a host endpoint, regardless of where the traffic is going - Egress rules apply only to traffic that is sent from the host endpoint (not a local workload) Finally, you must add an **applyOnForward: true expression** for a **doNotTrack policy** to work. ## Before you begin...[​](#before-you-begin) Before creating a **doNotTrack** network policy, read this [blog](https://www.tigera.io/blog/when-linux-conntrack-is-no-longer-your-friend/) to understand use cases, benefits, and trade offs. ## How to[​](#how-to) ### Bypass connection traffic for high connection server[​](#bypass-connection-traffic-for-high-connection-server) In the following example, a memcached server pod with **hostNetwork: true** was scheduled on the node memcached-node-1. We create a HostEndpoint for the node. Next, we create a GlobalNetwork Policy with symmetrical rules for ingress and egress with doNotTrack and applyOnForward set to true. ```yaml apiVersion: projectcalico.org/v3 kind: HostEndpoint metadata: name: memcached-node-1-eth0 labels: memcached: server spec: interfaceName: eth0 node: memcached-node-1 expectedIPs: - 10.128.0.162 --- apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: memcached-server spec: selector: memcached == 'server' applyOnForward: true doNotTrack: true ingress: - action: Allow protocol: TCP source: selector: memcached == 'client' destination: ports: - 12211 egress: - action: Allow protocol: TCP source: ports: - 12211 destination: selector: memcached == 'client' ``` ## Additional resources[​](#additional-resources) [Global network policy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) ### Defend against DoS attacks ## Big picture[​](#big-picture) Calico automatically enforces specific types of deny-list policies at the earliest possible point in the packet processing pipeline, including offloading to NIC hardware whenever possible. ## Value[​](#value) During a DoS attack, a cluster can receive massive numbers of connection requests from attackers. The faster these connection requests are dropped, the less flooding and overloading to your hosts. When you define DoS mitigation rules in Calico network policy, Calico enforces the rules as efficiently as possible to minimize the impact. ## Concepts[​](#concepts) ### Earliest packet processing[​](#earliest-packet-processing) The earliest point in the packet processing pipeline that packets can be dropped, depends on the Linux kernel version and the capabilities of the NIC driver and NIC hardware. Calico automatically uses the fastest available option. | Processed by... | Used by Calico if... | Performance | | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | | NIC hardware | The NIC supports **XDP offload** mode. | Fastest | | NIC driver | The NIC driver supports **XDP native** mode. | Faster | | Kernel | The kernel supports **XDP generic mode** and Calico is configured to explicitly use it. This mode is rarely used and has no performance benefits over iptables raw mode below. To enable, see [Felix Configuration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig). | Fast | | Kernel | If none of the modes above are available, **iptables raw** mode is used. | Fast | > **SECONDARY:** XDP modes require Linux kernel v4.16 or later. ## How to[​](#how-to) The high-level steps to defend against a DoS attack are: - [Step 1: Create host endpoints](#step-1-create-host-endpoints) - [Step 2: Add CIDRs to deny-list in a global network set](#step-2-add-cidrs-to-deny-list-in-a-global-network-set) - [Step 3: Create deny incoming traffic global network policy](#step-3-create-deny-incoming-traffic-global-network-policy) ### Best practice[​](#best-practice) The following steps walk through the above required steps, assuming no prior configuration is in place. A best practice is to proactively do these steps before an attack (create the host endpoints, network policy, and global network set). In the event of a DoS attack, you can quickly respond by just adding the CIDRs that you want to deny-list to the global network set. ### Step 1: Create host endpoints[​](#step-1-create-host-endpoints) First, you create the HostEndpoints corresponding to the network interfaces where you want to enforce DoS mitigation rules. In the following example, the HostEndpoint secures the interface named **eth0** with IP **10.0.0.1** on node **jasper**. ```yaml apiVersion: projectcalico.org/v3 kind: HostEndpoint metadata: name: production-host labels: apply-dos-mitigation: 'true' spec: interfaceName: eth0 node: jasper expectedIPs: ['10.0.0.1'] ``` ### Step 2: Add CIDRs to deny-list in a global network set[​](#step-2-add-cidrs-to-deny-list-in-a-global-network-set) Next, you create a Calico **GlobalNetworkset**, adding the CIDRs that you want to deny-list. In the following example, the global network set deny-lists the CIDR ranges **1.2.3.4/32** and **5.6.0.0/16**: ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkSet metadata: name: dos-mitigation labels: dos-deny-list: 'true' spec: nets: - '1.2.3.4/32' - '5.6.0.0/16' ``` ### Step 3: Create deny incoming traffic global network policy[​](#step-3-create-deny-incoming-traffic-global-network-policy) Finally, create a Calico GlobalNetworkPolicy adding the GlobalNetworkSet label (**dos-deny-list** in the previous step) as a selector to deny ingress traffic. To more quickly enforce the denial of forwarded traffic to the host at the packet level, use the **doNotTrack** and **applyOnForward** options. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: dos-mitigation spec: selector: apply-dos-mitigation == 'true' doNotTrack: true applyOnForward: true types: - Ingress ingress: - action: Deny source: selector: dos-deny-list == 'true' ``` ## Additional resources[​](#additional-resources) - [Global network sets](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkset) - [Global network policy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) - [Create a host endpoint](https://docs.tigera.io/calico-cloud/reference/resources/hostendpoint) - [Introduction to XDP](https://www.iovisor.org/technology/xdp) - [Advanced XDP documentation](https://prototype-kernel.readthedocs.io/en/latest/networking/XDP/index.html) --- ## Observability ### Observability and troubleshooting See what's going on in your cluster with network observability tools and detailed logging. ## Getting started[​](#getting-started) ##### [Manage alerts](https://docs.tigera.io/calico-cloud/observability/alerts) [Manage alerts and events for Calico Enterprise features.](https://docs.tigera.io/calico-cloud/observability/alerts) ##### [Kibana dashboards and logs](https://docs.tigera.io/calico-cloud/observability/kibana) [Learn the basics of using Elasticsearch logs and Kibana to gain visibility and troubleshoot.](https://docs.tigera.io/calico-cloud/observability/kibana) ##### [Packet capture](https://docs.tigera.io/calico-cloud/observability/packetcapture) [Capture live traffic for debugging microservices and application interaction.](https://docs.tigera.io/calico-cloud/observability/packetcapture) ##### [Visualize traffic to and from a cluster](https://docs.tigera.io/calico-cloud/observability/visualize-traffic) [Learn the power of network sets.](https://docs.tigera.io/calico-cloud/observability/visualize-traffic) ## Getting started with logs[​](#getting-started-with-logs) ##### [Overview](https://docs.tigera.io/calico-cloud/observability/elastic/overview) [Summary of the out-of-box features for Calico Cloud logs.](https://docs.tigera.io/calico-cloud/observability/elastic/overview) ##### [Archive logs](https://docs.tigera.io/calico-cloud/observability/elastic/archive-storage) [Archive logs to Syslog, Splunk, or Amazon S3 for maintaining compliance data.](https://docs.tigera.io/calico-cloud/observability/elastic/archive-storage) ##### [Overview](https://docs.tigera.io/calico-cloud/observability/elastic/overview) [Summary of the out-of-box features for Calico Cloud logs.](https://docs.tigera.io/calico-cloud/observability/elastic/overview) ##### [BGP logs](https://docs.tigera.io/calico-cloud/observability/elastic/bgp) [Key/value pairs of BGP activity logs and how to construct queries.](https://docs.tigera.io/calico-cloud/observability/elastic/bgp) ##### [Audit logs](https://docs.tigera.io/calico-cloud/observability/elastic/audit-overview) [Calico Cloud audit logs provide data on changes to resources.](https://docs.tigera.io/calico-cloud/observability/elastic/audit-overview) ## Flow logs[​](#flow-logs) ##### [Flow log data types](https://docs.tigera.io/calico-cloud/observability/elastic/flow/datatypes) [Data that Calico Cloud sends to Elasticsearch.](https://docs.tigera.io/calico-cloud/observability/elastic/flow/datatypes) ##### [Filter flow logs](https://docs.tigera.io/calico-cloud/observability/elastic/flow/filtering) [Filter Calico Cloud flow logs.](https://docs.tigera.io/calico-cloud/observability/elastic/flow/filtering) ##### [Configure flow log aggregation](https://docs.tigera.io/calico-cloud/observability/elastic/flow/aggregation) [Configure flow log aggregation to reduce log volume and costs.](https://docs.tigera.io/calico-cloud/observability/elastic/flow/aggregation) ##### [Enable HostEndpoint reporting in flow logs](https://docs.tigera.io/calico-cloud/observability/elastic/flow/hep) [Enable hostendpoint reporting in flow logs.](https://docs.tigera.io/calico-cloud/observability/elastic/flow/hep) ##### [Enable process-level information in flow logs](https://docs.tigera.io/calico-cloud/observability/elastic/flow/processpath) [Get visibility into process-level network activity in flow logs.](https://docs.tigera.io/calico-cloud/observability/elastic/flow/processpath) ##### [Enabling TCP socket stats in flow logs](https://docs.tigera.io/calico-cloud/observability/elastic/flow/tcpstats) [Enabling TCP socket stats information in flow logs](https://docs.tigera.io/calico-cloud/observability/elastic/flow/tcpstats) ## DNS logs[​](#dns-logs) ##### [Query DNS logs](https://docs.tigera.io/calico-cloud/observability/elastic/dns/dns-logs) [Key/value pairs of DNS activity logs and how to construct queries.](https://docs.tigera.io/calico-cloud/observability/elastic/dns/dns-logs) ##### [Filter DNS logs](https://docs.tigera.io/calico-cloud/observability/elastic/dns/filtering-dns) [Suppress DNS logs of low significance using filters.](https://docs.tigera.io/calico-cloud/observability/elastic/dns/filtering-dns) ## L7 logs[​](#l7-logs) ##### [Configure L7 logs](https://docs.tigera.io/calico-cloud/observability/elastic/l7/configure) [Configure and aggregate L7 logs.](https://docs.tigera.io/calico-cloud/observability/elastic/l7/configure) ##### [L7 log data types](https://docs.tigera.io/calico-cloud/observability/elastic/l7/datatypes) [L7 data that Calico Cloud sends to Elasticsearch.](https://docs.tigera.io/calico-cloud/observability/elastic/l7/datatypes) ### Manage alerts ## Big picture[​](#big-picture) Manage alerts and alert events for Calico Cloud features. ## Value[​](#value) You can configure alerts for many Calico Cloud features. Alerts are critical to teams for different reasons, for example: - **Visibility and troubleshooting** - alerts may indicate infrastructure problems, application bugs, or performance degradation - **Security** - alerts on suspicious traffic or workload behavior may indicate a compromise or malicious actor You can manage alerts and alert events in the web console, or using the CLI. Calico Cloud also provides alert templates for common tasks that you can rename and edit to suit your own needs. ## Before you begin[​](#before-you-begin) **Recommended** We recommend turning down the aggregation level for flow logs to ensure that you see pod-specific results. Calico Cloud aggregates flow logs over the external IPs for allowed traffic, and alert events will not provide pod-specific results (unless the traffic is denied by policy). > **WARNING:** Turning down aggregation levels for flow logs increases the amount of log data generated and may increase your Calico Cloud bill. To turn down aggregation on flow logs, go to [FelixConfiguration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig) and set the field, **flowLogsFileAggregationKindForAllowed** to **1**. ## How to[​](#how-to) - [Manage alerts in the web console](#manage-alerts-in-manager-ui) - [Manage alerts using CLI](#manage-alerts-using-cli) ### Manage alerts in the web console[​](#manage-alerts-in-the-web-console) You can view alert events in the web console in several places: the **Alerts** page, **Service Graph**, and the **Kibana** dashboard. Click **Activity**, **Alerts** to follow along. **Alerts page** The Alerts page lists **alert events** that are generated by alerts that you’ve configured. (A list of Alerts can be found by clicking the **Alert configuration** icon). ![alerts-list](https://docs.tigera.io/assets/images/alerts-list-c895990b8c87fca2e2b66bd7fd76bc8a.png) You can create alerts for many Calico Cloud features. Although the following list of features is not exhaustive and will grow, you get a sense of the range of alerts that can be displayed on this page. - Calico Cloud logs from Elasticsearch (flow, dns, audit, bgp, L7) - Deep packet inspection (DPI) - Threat defense (suspicious IPs, suspicious domains) - Web Application Firewall (WAF) Note the following: - The alert event list will be empty, if no alerts have occurred yet - You can dismiss alert events from view using the checkboxes or bulk action - The list may contain alert events that are identical or nearly identical. For nearly identical events, you can see differences in the `record` field when you expand the event. - Because alert events share the same interface, fields that do not apply to the alert are noted by “N/A” - You can filter alert events by Type. ![filter-alerts](https://docs.tigera.io/assets/images/filter-alerts-eaa2fe3322763084bebd2b94e76c092e.png) Note these types: - **Custom** - filters legacy global alert events that were created before v3.12 - **Global Alert** - includes alerts for Calico Cloud Elasticsearch logs (audit, dns, flow, L7, WAF) **Add/edit/delete alerts** To manage alerts, click the **Alerts Configuration** icon. The following alert is an example of a global alert in the list view. This sample alert generates alert events when there are 100 flows in the cluster in the last 5 mins. (The YAML version of this alert is shown in the section on using the [CLI](#examples).) ![alert-list-view](https://docs.tigera.io/assets/images/alert-list-view-3f7bc604187aafd65d74700cc1f7d4cd.png) To create a new alert, click the **New** drop-down menu, and select **Blank**. Global alerts use a domain-specific query language to select records from a data set to use in the alert. You can also select/omit specific namespaces. ![alert-example-ui](https://docs.tigera.io/assets/images/alert-example-ui-fbcafee8baad54340b7c1d8aa589d910.png) For help with fields on this page, see [GlobalAlert](https://docs.tigera.io/calico-cloud/reference/resources/globalalert). **Alert templates** From the **New** drop-down menu, select **Template**. ![alert-template](https://docs.tigera.io/assets/images/alert-template-7c10f275d34935a4ed83db08fa6a9f80.png) The template list contains alerts for common tasks created by Calico Cloud. With templates you can: - Update and rename an existing template - Create a new template from scratch - Create a new alert and save it as a template ### Manage alerts using CLI[​](#manage-alerts-using-cli) This section provides examples of how to create and delete global alerts using `kubectl` and YAML files. **Create a global alert** 1. Create a YAML file with one or more alerts. 2. Apply the alert to your cluster. ```bash kubectl apply -f ``` 3. Wait until the alert runs and check the status. ```bash kubectl get globalalert -o yaml ``` 4. In the web console, go to the **Alerts** page to view alert events. ### Examples[​](#examples) The following alert generates alert events when there are 100 flows in the cluster in the last 5 mins. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalAlert metadata: name: example-flows spec: description: '100 flows Example' summary: 'Flows example ${count} > 100' severity: 100 dataSet: flows metric: count condition: gt threshold: 100 ``` The following alert generates alert events when there is ssh traffic in the default namespace. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalAlert metadata: name: network.ssh spec: description: 'ssh flows to default namespace' summary: '[flows] ssh flow in default namespace detected from ${source_namespace}/${source_name_aggr}' severity: 100 period: 10m lookback: 10m dataSet: flows query: proto='tcp' AND action='allow' AND dest_port='22' AND (source_namespace='default' OR dest_namespace='default') AND reporter=src aggregateBy: [source_namespace, source_name_aggr] field: num_flows metric: sum condition: gt threshold: 0 ``` The following alert generates alert events when Calico Cloud globalnetworksets are modified. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalAlert metadata: name: policy.globalnetworkset spec: description: 'Changed globalnetworkset' summary: '[audit] [privileged access] change detected for ${objectRef.resource} ${objectRef.name}' severity: 100 period: 10m lookback: 10m dataSet: audit query: (verb=create OR verb=update OR verb=delete OR verb=patch) AND "objectRef.resource"=globalnetworksets aggregateBy: [objectRef.resource, objectRef.name] metric: count condition: gt threshold: 0 ``` The following alert generates alert events for all flow from processes in the data set. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalAlert metadata: name: example-process-set-embedded spec: description: Generate alerts for all flows from processes in the set summary: Generate alerts for all flows from processes in the set severity: 100 dataSet: flows query: process_name IN {"python?", "*checkoutservice"} ``` The following example generates alert events for DNS lookups that are not in the allowed domain set. Because this set can be potentially large, a variable is used in the query string and is referenced in the substitutions list. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalAlert metadata: name: example-domain-set-variable spec: description: Generate alerts for all DNS lookups not in the domain set summary: Generate alerts for all DNS lookups not in the domain set with variable severity: 100 dataSet: dns query: qname NOTIN ${domains} substitutions: - name: domains values: - '*cluster.local' - '?.mydomain.com' ``` **Delete a global alert** To delete a global alert and stop all alert event generation, use the following command. ```bash kubectl delete globalalert ``` ## Additional resources[​](#additional-resources) - [GlobalAlert and templates](https://docs.tigera.io/calico-cloud/reference/resources/globalalert) - Alerts for [Deep packet inspection](https://docs.tigera.io/calico-cloud/threat/deeppacketinspection) - Alerts for [suspicious IPs](https://docs.tigera.io/calico-cloud/threat/suspicious-ips) - Alerts for [suspicious domains](https://docs.tigera.io/calico-cloud/threat/suspicious-domains) - Alerts for [Web Application Firewall](https://docs.tigera.io/calico-cloud/threat/web-application-firewall) ### Dashboards Dashboards help you see what's going on in your cluster. See how your cluster is performing and visualize your system's log data. ## About dashboards[​](#about-dashboards) Calico Cloud provides a set of dashboards to help you understand the activity in your cluster. Each dashboard is made up of graphs, charts, and diagrams that visually represent the data in your logs. To view your dashboards, sign in to the web console and click the **Dashboards** icon. You can also create your own arrangement by creating a custom dashboard. With a custom dashboard, you can combine and arrange cards from any of the other dashboards. Creating custom dashboards is limited to users with Owner, Admin, or Dashboards Admin permissions. ## Role access to dashboards[​](#role-access-to-dashboards) The following user roles have access to all information in an organization's dashboards: - Owner - Admin - Dashboards Admin - Devops - Security - Viewer Users must be assigned one of the standard roles above, or a custom role with dashboard permissions, to access dashboards. Administrators can limit dashboard access for custom roles, for example to specific clusters or namespaces in a cluster. ### Cluster Health[​](#cluster-health) The **Cluster Health** dashboard provides a birds-eye view of cluster activity. ![Cluster Health Dashboard](https://docs.tigera.io/img/calico-enterprise/dashboards.png) ### Traffic Volume[​](#traffic-volume) The **Traffic Volume** dashboard provides a high-level view of traffic in your cluster. ![Traffic Volume Dashboard](https://docs.tigera.io/img/calico-cloud/dashboard-data-volume.png) ### DNS Logs[​](#dns-logs) The **DNS Logs** dashboard summarizes DNS data and logs into metrics, providing high-level information on the types of DNS lookups made, responses, and overall DNS performance. ![DNS Logs Dashboard](https://docs.tigera.io/img/calico-cloud/dashboard-dns-logs.png) ### Flow Logs[​](#flow-logs) The **Flow Logs** dashboard gives you an overview of how packets are being sent and received by all the pods in your cluster. Seeing this data helps you spot unusual flow activity, which may indicate a compromise. ![Flow Logs Dashboard](https://docs.tigera.io/img/calico-cloud/dashboard-flow-logs.png) ### HTTP Traffic[​](#http-traffic) The **HTTP Traffic** dashboard provides application performance metrics for inscope Kubernetes services. The data can assist service owners and platform personnel in assessing the health of cluster workloads without the need for a full service mesh. [L7 logs](https://docs.tigera.io/calico-cloud/observability/elastic/l7/configure) are not enabled by default, and must be configured. ![HTTP Traffic dashboard](https://docs.tigera.io/img/calico-cloud/dashboard-l7.png) ### Network Performance[​](#network-performance) The **Network Performance** dashboard provides TCP metrics to help you identify bottlenecks, packet loss, and performance issues. [Additional TCP statistics](https://docs.tigera.io/calico-cloud/observability/elastic/flow/tcpstats) such as Round Trip Time, Retransmission and Packet Loss are not enabled by default, and must be configured. ![Network Performance Dashboard](https://docs.tigera.io/img/calico-cloud/dashboard-tcp-performance.png) ### Create a custom dashboard > **SECONDARY:** Creating custom dashboards is a tech preview feature. Tech preview features are subject to significant changes before they become GA. You can customize your dashboard views by importing and modifying cards from the standard dashboards. You can also build cards from scratch using any fields from your flow, DNS, and L7 logs. ![Screenshot of Calico Cloud web console with view of custom dashboard.](https://docs.tigera.io/img/calico-cloud/custom-dashboard.png) ## Prerequisites[​](#prerequisites) - You are signed in to the web console as a user with the Owner, Admin, or Dashboards Admin role. ## Create a new dashboard[​](#create-a-new-dashboard) You can create your own custom dashboard by importing cards from other standard and custom dashboards. 1. From the web console, click **Dashboards > New Dashboard**. 2. Enter a name for your custom dashboard, and then press **Enter**. 3. To import cards from existing dashboards, click **Import Cards**. In the dialog that appears, select the cards you want, and then click **Import**. The page now displays the cards you selected. 4. Add, delete, or move the cards to create the arrangement that suits your needs. - To add a card, click **Actions > Import**, and then select a card. - To delete a card, click the ellipsis icon on the card, and then click **Delete**. - To export data from a card as a CSV file, click the ellipsis icon on the card, and then click **Export**. ## Modify an existing card[​](#modify-an-existing-card) ![Screenshot of Calico Cloud web console with view of edit card page.](https://docs.tigera.io/img/calico-cloud/edit-card.png) You can quickly edit any card on a custom dashboard to display the information you need. Even if you’re not creating a card from scratch, you can adjust queries, filters, or visual settings to refine the data. 1. From a custom dashboard, locate the card you want to modify. On the card, click the ellipsis icon , and then click **Edit**. 2. On the **Edit Card** screen, you can change the title, add a filter, change the chart type, or make any other modifications you want. The live chart view is updated in real time. 3. When you have finished making changes, click **Save**. The modified chart now appears on your custom dashboard page. ## Create a new card[​](#create-a-new-card) You can create a card by defining a query and selecting how the results are displayed. This lets you create custom data visualizations based on your specific needs. ***Prerequisites*** - You have a working knowledge of databases, search queries, and structured data analysis. ***Procedure*** 1. From a custom dashboard page, go to **Actions > Add New Card**. 2. Build your card selecting a data collection and chart type. Depending on the chart type, you'll be prompted to specify fields for the **Group By** and **Aggregation** sections. 3. When you have finished making changes, click **Save**. The modified chart now appears on your custom dashboard page. ### Kibana dashboards and logs > **WARNING:** Kibana dashboards are deprecated and will be removed in an upcoming release. During the deprecation period, you will have read-only access to Kibana dashboards. You can still [create custom dashboards](https://docs.tigera.io/calico-cloud/observability/create-custom-dashboard) using Calico Cloud's built-in dashboards. ## Kibana[​](#kibana) Kibana is the frontend for Calico Cloud Elasticsearch, which is the logging infrastructure that centrally stores logs from all managed clusters. Kibana provides an interface to explore Elasticsearch logs and gain insights into workload communication traffic volume, performance, and other key aspects of cluster operations. Log data is also summarized in custom dashboards. The following logs are generated by Calico Cloud. All logs are enabled by default except **l7 logs**, which must be explicitly enabled. | Log type | **Description** | Index in Kibana | | -------- | ---------------------------------------------------------------------------------------------------- | ---------------------------- | | flow | Layer 3/4 network flows for workloads: source and destination namespaces, pods, labels, and policies | tigera\_secure\_ee\_flows\* | | l7 | Layer 7 network flows for workloads | tigera\_secure\_ee\_l7\* | | audit | Audit logs for Calico Cloud resources | tigera\_secure\_ee\_audit\* | | bgp | Calico Cloud networking BGP peering and route propagation. | tigera\_secure\_ee\_bgp.\* | | dns | DNS lookups and responses from Calico Cloud domain-based policy. | tigera\_secure\_ee\_dns\* | | events | Calico Cloud intrusion detection events: suspicious IPs, suspicious domains, and global alerts | tigera\_secure\_ee\_events\* | ## Start Kibana and access dashboards[​](#start-kibana-and-access-dashboards) In the web console, from the left navbar select, **Kibana**. A new browser tab opens into Kibana. In Kibana, click the hamburger icon in the top left corner, and select **Analytics**, **Dashboard**. ![kibana-dashboard](https://docs.tigera.io/assets/images/kibana-dashboard-d9381464d517d33e2a99400ca8a3e048.png) A list of curated dashboards is displayed. Note that some log types do not have a default dashboard (`bgp` and `events`). ### DNS dashboard[​](#dns-dashboard) ![dns-dashboard](https://docs.tigera.io/assets/images/dns-dashboard-104f543e8fc607107d780a0ef29e093b.png) The DNS dashboard summarizes DNS data and logs into metrics, providing high-level information on the types of DNS lookups made, responses, and overall DNS performance. By default, DNS activity logs are captured only for requests/responses from Kubernetes built-in DNS services (CoreDNS). DNS activity to an external DNS server can be captured by configuring the parameter, `dnsTrustedServers` in [Felix](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig). DNS activity to Node local server is not supported. The dashboard provides the following metrics/data, which can be edited as required. | Metric/data | Description | | ------------------------------- | ------------------------------------------------------------------- | | DNS total requests | Cumulative DNS requests over the reporting period. Default: 24hrs. | | DNS requests | Type of DNS request. | | DNS responses | DNS response codes which may indicate issues with specific lookups. | | DNS Top 10 external domains | Count of top domains in lookups. | | DNS internal query | Lookups within the Kubernetes cluster. | | DNS external query | Lookups to non-cluster domains. | | DNS Latency | Measured latency which can indicate DNS issues. | | DNS internal queries by service | Top types of requests within the cluster per service. | | DNS external queries by service | Top types of requests external to the cluster per service. | | DNS response code by service | Top DNS response codes per client. | | DNS query count by server | Volume of DNS traffic per DNS server. | | DNS transfer by service | Volume of DNS traffic per service. | | DNS logs | Raw DNS logs. | ### L7 HTTP dashboard[​](#l7-http-dashboard) ![l7-dashboard](https://docs.tigera.io/assets/images/l7-dashboard-c9ce16b14c131a58e2fe90ed9d05943a.png) The L7 HTTP dashboard provides application performance metrics for inscope Kubernetes services. The data can assist service owners and platform personnel in assessing the health of cluster workloads without the need for a full service mesh. [L7 logs](https://docs.tigera.io/calico-cloud/observability/elastic/l7/configure) are not enabled by default, and must be configured. The default metrics are: - L7 HTTP requests - L7 all services - L7 HTTP duration - L7 HTTP methods - L7 HTTP response codes - L7 HTTP request duration - L7 HTTP requests over time - L7 HTTP method by service - L7 HTTP response by service - L7 HTTP bytes transferred - L7 Top URLs - L7-search (raw HTTP logs) ### Tigera Secure EE audit logs dashboard[​](#tigera-secure-ee-audit-logs-dashboard) ![audit-logs-dashboard](https://docs.tigera.io/assets/images/audit-logs-dashboard-756d35c1965e6982ab756e5fa38098d1.png) The Tigera Secure EE audit logs dashboard provides historical events of changes made to your deployment. These events can be used to understand updates to resources, privileged access and actions, and can also help demonstrate compliance for different regulatory concerns. Audit logs listed in the section, `audit-search` can be expanded by clicking on the triangular expand icon, which presents the log in Table format by default. Clicking on JSON in the Expanded document displays the same log in JSON format. The logs can be filtered in the Audit Filtering Controls. ### Tigera Secure EE flow logs dashboard[​](#tigera-secure-ee-flow-logs-dashboard) ![flow-logs-dashboard](https://docs.tigera.io/assets/images/flow-logs-dashboard-93c25fcfdabdf7535b5752ab0587238f.png) The Tigera Secure EE flow Logs dashboard lets you analyze flow logs using the filter options in the Flow Filtering window. The flow logs matching the applied filter are displayed below in the flow logs window. To review a specific flow log in detail, click the triangular expand icon to the left of the flow. The full flow log is now displayed in Tabular format by default. To view the log in JSON format click the JSON header. ![flow-logs-dashboard](https://docs.tigera.io/assets/images/flow-logs-json-ab0b57f65034668b187f018b21484356.png) ### Tigera Secure EE Tor-VPN logs[​](#tigera-secure-ee-tor-vpn-logs) ![tor-vpn-dashboard](https://docs.tigera.io/assets/images/tor-vpn-dashboard-85159b404ee9483280f31d1894cca97d.png) Tor and VPN-based traffic indicate the use of anonymization techniques in an attempt to mask the origins and destination of network traffic. Calico Cloud has built-in capabilities to assist with detecting such traffic and requires minimal [configuration](https://docs.tigera.io/calico-cloud/threat/tor-vpn-feed-and-dashboard) to activate. Once enabled, the Tigera Secure EE Tor-VPN logs dashboard can provide a view into any traffic to/from Tor and VPN gateways. The information quickly provides InfoSec teams and operators a focused view on anonymization-based traffic patterns. The reported flows can be filtered in the Tor-VPN controls window and the flow logs for inscope traffic can be reviewed in the Tor-VPN-search window. ## Create custom filters and queries[​](#create-custom-filters-and-queries) Each dashboard has advanced filtering options if pre-built dashboards are insufficient. For example: - To build a query from all fields available in the logs, click **Add Filter** - To create a manual query, click **Search** (next to the disk icon on the left). The following example shows a query `process_name :*curl*` for the `process_name field` matching glob pattern, *curl*. Only logs where `field process_name` contains the string `curl` are filtered. ![custom-search](https://docs.tigera.io/assets/images/custom-search-1a78fc0efe196469b731c119c72ef580.png) ## View logs by indices[​](#view-logs-by-indices) To view logs by indices, click the hamburger menu, select **Analytics**, and click **Discover**. ![all-flow-logs](https://docs.tigera.io/assets/images/all-flow-logs-8ec4a91b8a89bd0a177f4d78c016b222.png) ### Packet capture ## Big picture[​](#big-picture) Capture live traffic inside a Kubernetes cluster, and export to visualization tools like Wireshark for troubleshooting and debugging applications. ## Value[​](#value) Calico Cloud packet capture is implemented in a Kubernetes-native way so you can troubleshoot service/application connectivity issues and performance issues. You can start a packet capture in the web console Service Graph, or using the CLI. Packet capture integration with **Service Graph** makes it very easy to capture traffic for a specific namespace, service, replica set, daemonset, statefulset, or pod. Just right-click on an endpoint to start or schedule a capture, and then download capture files to your favorite visualization tool like WireShark. With Calico Cloud packet capture you can: - Run packet capture whenever you want (available 24/7) - Preschedule packet captures to start and stop when needed - Customize packet captures by port and protocol - Share packet capture jobs **Demos and blogs** - [Video: packet capture demo](https://www.tigera.io/features/packet-capture/) - [Troubleshooting microservices with Dynamic Packet Capture](https://thenewstack.io/faster-troubleshooting-with-dynamic-packet-capture/) ## Concepts[​](#concepts) ## About packet capture[​](#about-packet-capture) Typically, when you troubleshoot microservices and applications for connectivity issues or slow performance, you run a traditional packet capture tool like **tcpdump** against a container in a pod. But live troubleshooting in an ephemeral Kubernetes environment is tricky; problems do not last a long time, and happen randomly. So you need to be very fast to capture meaningful information to determine root causes. Calico Cloud makes it easy with these basic steps: 1. Determine the workload(s) you want to capture. 2. Start/schedule a packet capture job in Service Graph (Manager UI) or the CLI. 3. After the capture is finished, download the packet capture files (known as `pcap` files), and import them into your analysis tool (for example, WireShark). For a simple use case workflow, see [Faster troubleshooting of microservices, containers, and Kubernetes with Dynamic Packet Capture](https://www.tigera.io/blog/faster-troubleshooting-of-microservices-containers-and-kubernetes-with-dynamic-packet-capture/). ## Before you begin[​](#before-you-begin) **Not supported** - Capturing traffic from host networked pods or host endpoints - Capturing traffic from pods with multiple interfaces - Capturing traffic for pods running on Windows hosts ## How to[​](#how-to) - [Enable packet capture](#enable-packet-capture) - [Packet capture in Service Graph](#packet-capture-in-service-graph) - [Packet capture using the command line](#packet-capture-using-the-command-line) - [Store and rotate capture files](#store-and-rotate-capture-files) - [Enforce RBAC for capture tasks for CLI users](#enforce-rbac-for-capture-tasks-for-cli-users) ### Enable packet capture[​](#enable-packet-capture) ##### Enable packet capture using kubectl[​](#enable-packet-capture-using-kubectl) Create a PacketCaptureAPI custom resource, named `tigera-secure` in the cluster. ```bash kubectl apply -f - < ``` **Schedule a packet capture job** You can schedule a packet capture job to start and/or stop at a specific time using RFC3339 format. In the following example, a traffic capture job is scheduled for 10 minutes, between 00:30 UTC and 00:40 UTC for all pods in the sample namespace. ```yaml apiVersion: projectcalico.org/v3 kind: PacketCapture metadata: name: sample-capture-all namespace: sample spec: selector: all() startTime: '2021-09-08T00:30:00Z' endTime: '2021-09-08T00:40:00Z' ``` **Monitor status of packet capture job** After you start capture a job, it cycles through these states: Scheduled (if applicable), WaitingForTraffic, Capturing, and Finished. To monitor the status of a PacketCapture, use the following command: ```bash kubectl get packetcaptures -A ``` **Stop a packet capture job** To stop a capture job immediately, update the PacketCaptureResource by setting the `endTime` to the current time (or earlier). **Stop a packet capture job, and delete the capture file from the cluster** ```bash kubectl delete -f ``` **Delete a packet capture job** ```bash kubectl delete -f ``` **Find packet capture files** To find generated capture files, query the status of the PacketCapture: ```bash kubectl get packetcaptures -n -o yaml ``` ```bash export NS= export NAME= ``` **Sample output** ```yaml apiVersion: projectcalico.org/v3 kind: PacketCapture metadata: name: sample-capture-all namespace: sample spec: selector: all() status: files: - directory: /var/log/calico/pcap fileNames: - pod_cali.pcap node: node-0 state: Capturing ``` **Get packet capture files from pods** Get the pod on the node with the packet capture that you want. ```bash kubectl get pods -n tigera-fluentd --no-headers --field-selector spec.nodeName="" ``` Copy the packet capture using the pod information. ```bash kubectl cp tigera-fluentd/:var/log/calico/pcap/sample/sample-capture/ . ``` **Delete packet capture files** ```bash kubectl exec -it tigera-fluentd/ -- sh -c "rm -r /var/log/calico/pcap/sample/sample-capture/" ``` ### Store and rotate capture files[​](#store-and-rotate-capture-files) Packet capture files are stored on the host-mounted volume used for calico nodes. FelixConfig contains several parameters for storing and rotating capture files. **Note**: - Capture files are stored using the following directory structure: `{namespace}/{packet capture resource name}` - The active packet capture file is identified using the following schema: `{workload endpoint name}_{host network interface}.pcap` - Rotated capture file names contain an index matching the rotation timestamp - Packet capture files are deleted after the packet capture resource is deleted. **Rotate capture files** The Felix parameter, `captureRotationSeconds` lets you schedule how often saved pcap are rotated. In the following example, the time rotation time is one day. ```bash kubectl patch felixconfiguration default -p '{"spec":{"captureRotationSeconds":"86400"}}' ``` ### Enforce RBAC for capture tasks for CLI users[​](#enforce-rbac-for-capture-tasks-for-cli-users) Packet capture permissions are enforced using the standard Kubernetes RBAC for CLI users, based on Role and RoleBindings within a namespace. **Example** The following Role and RoleBindings shows how to allow user jane to create/delete/get/list/update/watch packet captures for a specific namespace. ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: sample name: tigera-packet-capture-role rules: - apiGroups: ['projectcalico.org'] resources: ['packetcaptures'] verbs: ['get', 'list', 'watch', 'create', 'update', 'patch', 'delete'] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: tigera-packet-capture-role-jane namespace: sample subjects: - kind: ServiceAccount name: jane roleRef: kind: Role name: tigera-packet-capture-role apiGroup: rbac.authorization.k8s.io ``` To allow user jane to access (get and delete) the capture files generated for a specific namespace, a role/role binding similar to the one below can be used: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: tigera-authentication-clusterrole-jane rules: - apiGroups: ['projectcalico.org'] resources: ['authenticationreviews'] verbs: ['create'] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: tigera-authentication-clusterrolebinding-jane roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: tigera-authentication-clusterrole-jane subjects: - kind: ServiceAccount name: jane namespace: default --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: sample name: tigera-capture-files-role rules: - apiGroups: ['projectcalico.org'] resources: ['packetcaptures/files'] verbs: ['get', 'delete'] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: tigera-capture-files-role-jane namespace: sample subjects: - kind: ServiceAccount name: jane namespace: default roleRef: kind: Role name: tigera-capture-files-role apiGroup: rbac.authorization.k8s.io ``` ### Visualize traffic to and from a cluster With Service Graph, you know the value of seeing pod-to-pod traffic within your cluster. But what about traffic external to your cluster? To learn how to visualize traffic to and from a cluster, see [Get started with network sets](https://docs.tigera.io/calico-cloud/network-policy/networksets). ![google-networkset](https://docs.tigera.io/assets/images/google-networkset-1d5879d8426568a6dae6e11f905ef609.png) ### Manage Calico Cloud logs ## [📄️ Overview](https://docs.tigera.io/calico-cloud/observability/elastic/overview) [Summary of the out-of-box features for Calico Cloud logs.](https://docs.tigera.io/calico-cloud/observability/elastic/overview) ## [📄️ Archive logs](https://docs.tigera.io/calico-cloud/observability/elastic/archive-storage) [Archive logs to Syslog, Splunk, or Amazon S3 for maintaining compliance data.](https://docs.tigera.io/calico-cloud/observability/elastic/archive-storage) ## [🗃️ Flow logs](https://docs.tigera.io/calico-cloud/observability/elastic/flow/) [6 items](https://docs.tigera.io/calico-cloud/observability/elastic/flow/) ## [📄️ Audit logs](https://docs.tigera.io/calico-cloud/observability/elastic/audit-overview) [Calico Cloud audit logs provide data on changes to resources.](https://docs.tigera.io/calico-cloud/observability/elastic/audit-overview) ## [🗃️ DNS logs](https://docs.tigera.io/calico-cloud/observability/elastic/dns/) [2 items](https://docs.tigera.io/calico-cloud/observability/elastic/dns/) ## [📄️ BGP logs](https://docs.tigera.io/calico-cloud/observability/elastic/bgp) [Key/value pairs of BGP activity logs and how to construct queries.](https://docs.tigera.io/calico-cloud/observability/elastic/bgp) ## [🗃️ L7 logs](https://docs.tigera.io/calico-cloud/observability/elastic/l7/) [2 items](https://docs.tigera.io/calico-cloud/observability/elastic/l7/) ### Overview ## Big picture[​](#big-picture) Use Calico Cloud log data for visibility and troubleshooting Kubernetes clusters. ## Value[​](#value) Workloads and policies are highly dynamic. To troubleshoot Kubernetes clusters, you need logs with workload identity and context. Calico Cloud deploys an Elasticsearch cluster and Kibana instance during installation with these features: - Logs with workload context - Centralized log collection for multiple clusters for Calico Cloud multi-cluster-management - View Elasticsearch logs in the Calico Cloud web console (Kibana dashboard and Flow Visualizer), and the [Elasticsearch API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search.html) - Standard Kubernetes RBAC for granular access control to logs - Collect/archive logs or subset of logs - Log aggregation for high-volume logs - Configure data retention settings to manage cluster disk space - Integration with third-party tools like Amazon S3, Syslog, Splunk ## Concepts[​](#concepts) ### Logs types[​](#logs-types) Elasticsearch logs provide the visibility and troubleshooting backend for Calico Cloud. | Log type | Description | Log source | RBAC | Index | | -------- | ---------------------------------------------------------------------------------------------- | ------------------------------------------- | ------------ | ----------------------------- | | flow | Network flows for workloads: source and destination namespaces, pods, labels, and policies | Calico Cloud cnx-node (Felix) | `flows` | `tigera_secure_ee_flows` | | audit | Audit logs for Calico Cloud resources | Calico Cloud apiserver | `audit_ee` | `tigera_secure_ee_audit_ee` | | | Audit logs for Kubernetes resources | Kubernetes apiserver | `audit_kube` | `tigera_secure_ee_audit_kube` | | | | Both audit logs above | `audit*` | `tigera_secure_ee_audit*` | | bgp | Calico Cloud networking BGP peering and route propagation | Calico Cloud cnx-node (BIRD) | `ee_bgp` | `tigera_secure_ee_bgp.*` | | dns | DNS lookups and responses from Calico Cloud domain-based policy | Calico Cloud cnx-node (Felix) | `ee_dns` | `tigera_secure_ee_dns` | | ids | Calico Cloud intrusion detection events: suspicious IPs, suspicious domains, and global alerts | Calico Cloud intrusion-detection-controller | `ee_events` | `tigera_secure_ee_events` | > **SECONDARY:** Because of their high-volume, flow and dns logs support aggregation. ### Default log configuration and security[​](#default-log-configuration-and-security) Calico Cloud automatically installs fluentd on all nodes and collects flow, audit, and DNS logs. You can configure additional destinations like Amazon S3, Syslog, Splunk. Calico Cloud enables user authentication in Elasticsearch, and secures access to Elasticsearch and Kibana instances using network policy. ### RBAC and log access[​](#rbac-and-log-access) You control user access to logs using the standard Kubernetes RBAC cluster role and cluster role binding. For example: ```yaml kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: bob-es-access subjects: - kind: User name: bob apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: audit-ee-only apiGroup: rbac.authorization.k8s.io ``` You configure Elasticsearch log access per cluster using RBAC and the Kubernetes API group, `lma.tigera.io`. For example: ```yaml apiGroups: ['lma.tigera.io'] resources: ['app-cluster'] resourceNames: ['flows', 'dns'] verbs: ['get'] ``` ### Logs for compliance reporting[​](#logs-for-compliance-reporting) Calico Cloud compliance reports are based on archived **flow logs** and **audit logs** for these resources: - Pods - Host endpoints - Service accounts - Namespaces - Kubernetes service endpoints - Global network sets - Calico Cloud and Kubernetes network policies - Global network policies - Network sets Calico Cloud also supports archiving [Cloudwatch for EKS audit logs](https://docs.tigera.io/calico-cloud/reference/installation/api#logcollectorspec). ## Additional resources[​](#additional-resources) - [Configure flow log aggregation](https://docs.tigera.io/calico-cloud/observability/elastic/flow/aggregation) - [Audit logs](https://docs.tigera.io/calico-cloud/observability/elastic/audit-overview) - [BGP logs](https://docs.tigera.io/calico-cloud/observability/elastic/bgp) - [DNS logs](https://docs.tigera.io/calico-cloud/observability/elastic/dns/dns-logs) - [Archive logs](https://docs.tigera.io/calico-cloud/observability/elastic/archive-storage) - [Log collection options](https://docs.tigera.io/calico-cloud/reference/installation/api#logcollectorspec) ### Archive logs ## Big picture[​](#big-picture) Archive Calico Cloud logs to SIEMs like Syslog, Splunk, or Amazon S3 to meet compliance storage requirements. ## Value[​](#value) Archiving your Calico Cloud Elasticsearch logs to storage services like Amazon S3, Syslog, or Splunk are reliable options for maintaining and consolidating your compliance data long term. ## Before you begin[​](#before-you-begin) **Supported logs for export** - Syslog - flow, dns, idsevents, audit - Amazon S3 - l7, flow, dns, runtime, audit - Splunk - flow, audit, dns ## How to[​](#how-to) > **SECONDARY:** Because Calico Cloud and Kubernetes logs are integral to Calico Cloud diagnostics, there is no mechanism to tune down the verbosity. To manage log verbosity, filter logs using your SIEM. **Tab: Amazon S3** 1. Create an AWS bucket to store your logs. You will need the bucket name, region, key, secret key, and the path in the following steps. 2. Create a Secret in the `tigera-operator` namespace named `log-collector-s3-credentials` with the fields `key-id` and `key-secret`. Example: ```text kubectl create secret generic log-collector-s3-credentials \ --from-literal=key-id= \ --from-literal=key-secret= \ -n tigera-operator ``` 3. Update the [LogCollector](https://docs.tigera.io/calico-cloud/reference/installation/api#logcollector) resource named, `tigera-secure` to include an [S3 section](https://docs.tigera.io/calico-cloud/reference/installation/api#s3storespec) with your information noted from above. Example: ```yaml apiVersion: operator.tigera.io/v1 kind: LogCollector metadata: name: tigera-secure spec: additionalStores: s3: bucketName: bucketPath: region: ``` This can be done during installation by editing the custom-resources.yaml by applying it, or after installation by editing the resource with the command: ```bash kubectl edit logcollector tigera-secure ``` **Tab: Syslog** 1. Update the [LogCollector](https://docs.tigera.io/calico-cloud/reference/installation/api#logcollector) resource named `tigera-secure` to include a [Syslog section](https://docs.tigera.io/calico-cloud/reference/installation/api#syslogstorespec) with your syslog information. Example: ```yaml apiVersion: operator.tigera.io/v1 kind: LogCollector metadata: name: tigera-secure spec: additionalStores: syslog: # (Required) Syslog endpoint, in the format protocol://host:port endpoint: tcp://1.2.3.4:514 # (Optional) If messages are being truncated set this field packetSize: 1024 # (Required) Types of logs to forward to Syslog (must specify at least one option) logTypes: - Audit - DNS - Flows - IDSEvents ``` This can be done during installation by editing the custom-resources.yaml by applying it or after installation by editing the resource with the command: ```bash kubectl edit logcollector tigera-secure ``` 2. You can control which types of Calico Cloud log data you would like to send to syslog. The [Syslog section](https://docs.tigera.io/calico-cloud/reference/installation/api#syslogstorespec) contains a field called `logTypes` which allows you to list which log types you would like to include. The allowable log types are: - Audit - DNS - Flows - IDSEvents Refer to the [Syslog section](https://docs.tigera.io/calico-cloud/reference/installation/api#syslogstorespec) for more details on what data each log type represents. > **SECONDARY:** The log type `IDSEvents` is only supported for a cluster that has [LogStorage](https://docs.tigera.io/calico-cloud/reference/installation/api#logstorage) configured. It is because intrusion detection event data is pulled from the corresponding LogStorage datastore directly. The `logTypes` field is a required, which means you must specify at least one type of log to export to syslog. **TLS configuration** 3. You can enable TLS option for syslog forwarding by including the "encryption" option in the [Syslog section](https://docs.tigera.io/calico-cloud/reference/installation/api#syslogstorespec). ```yaml apiVersion: operator.tigera.io/v1 kind: LogCollector metadata: name: tigera-secure spec: additionalStores: syslog: # (Required) Syslog endpoint, in the format protocol://host:port endpoint: tcp://1.2.3.4:514 # (Optional) If messages are being truncated set this field packetSize: 1024 # (Optional) To Configure TLS mode encryption: TLS # (Required) Types of logs to forward to Syslog (must specify at least one option) logTypes: - Audit - DNS - Flows - IDSEvents ``` 4. Using the self-signed CA with the field name tls.crt, create a configmap in the tigera-operator namespace named, syslog-ca. Example: > **SECONDARY:** Skip this step if publicCA bundle is good enough to verify the server certificates. ```bash kubectl create configmap syslog-ca --from-file=tls.crt -n tigera-operator ``` **Tab: Splunk** **Support** In this release, only [Splunk Enterprise](https://www.splunk.com/en_us/products/splunk-enterprise.html) is supported. Calico Cloud uses Splunk's **HTTP Event Collector** to send data to Splunk server. To copy the flow, audit, and dns logs to Splunk, follow these steps: 1. Create a HTTP Event Collector token by following the steps listed in Splunk's documentation for your specific Splunk version. Here is the link to do this for [Splunk version 8.0.0](https://docs.splunk.com/Documentation/Splunk/8.0.0/Data/UsetheHTTPEventCollector). 2. Create a Secret in the `tigera-operator` namespace named `logcollector-splunk-credentials` with the field `token`. Example: ```text kubectl create secret generic logcollector-splunk-credentials \ --from-literal=token= \ -n tigera-operator ``` 3. Update the [LogCollector](https://docs.tigera.io/calico-cloud/reference/installation/api#logcollector) resource named `tigera-secure` to include a [Splunk section](https://docs.tigera.io/calico-cloud/reference/installation/api#splunkstorespec) with your Splunk information. Example: ```yaml apiVersion: operator.tigera.io/v1 kind: LogCollector metadata: name: tigera-secure spec: additionalStores: splunk: # Splunk HTTP Event Collector endpoint, in the format protocol://host:port endpoint: https://1.2.3.4:8088 ``` This can be done during installation by editing the custom-resources.yaml by applying it or after installation by editing the resource with the command: ```text kubectl edit logcollector tigera-secure ``` ### Configure flow logs ## [📄️ Flow log data types](https://docs.tigera.io/calico-cloud/observability/elastic/flow/datatypes) [Data that Calico Cloud sends to Elasticsearch.](https://docs.tigera.io/calico-cloud/observability/elastic/flow/datatypes) ## [📄️ Filter flow logs](https://docs.tigera.io/calico-cloud/observability/elastic/flow/filtering) [Filter Calico Cloud flow logs.](https://docs.tigera.io/calico-cloud/observability/elastic/flow/filtering) ## [📄️ Configure flow log aggregation](https://docs.tigera.io/calico-cloud/observability/elastic/flow/aggregation) [Configure flow log aggregation to reduce log volume and costs.](https://docs.tigera.io/calico-cloud/observability/elastic/flow/aggregation) ## [📄️ Enable HostEndpoint reporting in flow logs](https://docs.tigera.io/calico-cloud/observability/elastic/flow/hep) [Enable hostendpoint reporting in flow logs.](https://docs.tigera.io/calico-cloud/observability/elastic/flow/hep) ## [📄️ Enabling TCP socket stats in flow logs](https://docs.tigera.io/calico-cloud/observability/elastic/flow/tcpstats) [Enabling TCP socket stats information in flow logs](https://docs.tigera.io/calico-cloud/observability/elastic/flow/tcpstats) ## [📄️ Enable process-level information in flow logs](https://docs.tigera.io/calico-cloud/observability/elastic/flow/processpath) [Get visibility into process-level network activity in flow logs.](https://docs.tigera.io/calico-cloud/observability/elastic/flow/processpath) ### Flow log data types ## Big picture[​](#big-picture) Calico Cloud sends the following data to Elasticsearch. The following table details the key/value pairs in the JSON blob, including their [Elasticsearch datatype](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html). | Name | Datatype | Description | | --------------------------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `host` | keyword | Name of the node that collected the flow log entry. | | `start_time` | date | Start time of log collection in UNIX timestamp format. | | `end_time` | date | End time of log collection in UNIX timestamp format. | | `action` | keyword | - `allow`: Calico Cloud accepted the flow. - `deny`: Calico Cloud denied the flow. | | `bytes_in` | long | Number of incoming bytes since the last export. | | `bytes_out` | long | Number of outgoing bytes since the last export. | | `dest_ip` | ip | IP address of the destination pod. A null value indicates aggregation. | | `dest_name` | keyword | Contains one of the following values: - Name of the destination pod. - Name of the pod that was aggregated or the endpoint is not a pod. Check `dest_name_aggr` for more information, such as the name of the pod if it was aggregated. | | `dest_name_aggr` | keyword | Contains one of the following values: - Aggregated name of the destination pod. - `pvt`: endpoint is not a pod. Its IP address belongs to a private subnet. - `pub`: endpoint is not a pod. Its IP address does not belong to a private subnet. It is probably an endpoint on the public internet. | | `dest_namespace` | keyword | Namespace of the destination endpoint. A `-` means the endpoint is not namespaced. | | `dest_port` | long | Destination port. Not applicable for ICMP packets. | | `dest_service_name` | keyword | Name of the destination service. A `-` means the original destination did not correspond to a known Kubernetes service (e.g. a services ClusterIP). | | `dest_service_namespace` | keyword | Namespace of the destination service. A `-` means the original destination did not correspond to a known Kubernetes service (e.g. a services ClusterIP). | | `dest_service_port` | keyword | Port name of the destination service. A `-` means : - the original destination did not correspond to a known Kubernetes service (e.g. a services ClusterIP), or - the destination port is aggregated. A `*` means there are multiple service port names matching the destination port number. | | `dest_type` | keyword | Destination endpoint type. Possible values: - `wep`: A workload endpoint, a pod in Kubernetes. - `ns`: A Networkset. If multiple Networksets match, then the one with the longest prefix match is chosen. - `net`: A Network. The IP address did not fall into a known endpoint type. | | `dest_labels` | array of keywords | Labels applied to the destination pod. A hyphen indicates aggregation. | | `dest_domains` | array of keywords | Find all the destination domain names for use in a DNS policy by examining `dest_domains`. The field displays information on the top-level domains linked to the destination IP. Applies to flows reported from the source to destinations outside the cluster. If `flowLogsDestDomainsByClient` is disabled, having `dest_domains`: \["A"] doesn't guarantee that the flow corresponds to a connection with domain name A. The destination IP may also be linked to other domain names not yet captured by Calico. | | `reporter` | keyword | - `src`: flow came from the pod that initiated the connection. - `dst`: flow came from the pod that received the initial connection. - `fwd`: flow was forwarded through the node without being the source or destination. | | `num_flows` | long | Number of flows aggregated into this entry during this export interval. | | `num_flows_completed` | long | Number of flows that were completed during the export interval. | | `num_flows_started` | long | Number of flows that were started during the export interval. | | `num_process_names` | long | Number of unique process names aggregated into this entry during this export interval. | | `num_process_ids` | long | Number of unique process ids aggregated into this entry during this export interval. | | `num_process_args` | long | Number of unique process args aggregated into this entry during this export interval. | | `nat_outgoing_ports` | array of ints | List of [NAT](https://en.wikipedia.org/wiki/Network_address_translation) outgoing ports for the packets that were Source NAT'd in the flow | | `packets_in` | long | Number of incoming packets since the last export. | | `packets_out` | long | Number of outgoing packets since the last export. | | `proto` | keyword | Protocol. | | `policies` | array of arrays | List of policies that interacted with this flow. See [Format of the policies field](#format-of-the-policies-field). | | `process_name` | keyword | The name of the process that initiated or received the connection or connection request. This field will have the executable path if flowLogsCollectProcessPath is enabled. A "-" indicates that the process name is not logged. A "\*" indicates that the per flow process limit has exceeded and the process names are now aggregated. | | `process_id` | keyword | The process ID of the corresponding process (indicated by the `process_name` field) that initiated or received the connection or connection request. A "-" indicates that the process ID is not logged. A "\*" indicates that there are more than one unique process IDs for the corresponding process name. | | `process_args` | array of strings | The arguments with which the executable was invoked. The size of the list depends on the per flow process args limit. | | `source_ip` | ip | IP address of the source pod. A null value indicates aggregation. | | `source_name` | keyword | Contains one of the following values: - Name of the source pod. - Name of the pod that was aggregated or the endpoint is not a pod. Check `source_name_aggr` for more information, such as the name of the pod if it was aggregated. | | `source_name_aggr` | keyword | Contains one of the following values: - Aggregated name of the source pod. - `pvt`: Endpoint is not a pod. Its IP address belongs to a private subnet. - `pub`: the endpoint is not a pod. Its IP address does not belong to a private subnet. It is probably an endpoint on the public internet. | | `source_namespace` | keyword | Namespace of the source endpoint. A `-` means the endpoint is not namespaced. | | `source_port` | long | Source port. A null value indicates aggregation. | | `source_type` | keyword | The type of source endpoint. Possible values: - `wep`: A workload endpoint, a pod in Kubernetes. - `ns`: A Networkset. If multiple Networksets match, then the one with the longest prefix match is chosen. - `net`: A Network. The IP address did not fall into a known endpoint type. | | `source_labels` | array of keywords | Labels applied to the source pod. A hyphen indicates aggregation. | | `original_source_ips` | array of ips | List of external IP addresses collected from requests made to the cluster through an ingress resource. This field is only available if capturing external IP addresses is configured. | | `num_original_source_ips` | long | Number of unique external IP addresses collected from requests made to the cluster through an ingress resource. This count includes the IP addresses included in the `original_source_ips` field. This field is only available if capturing external IP addresses is configured. | | `tcp_mean_send_congestion_window` | long | Mean tcp send congestion window size. This field is only available if flowLogsEnableTcpStats is enabled | | `tcp_min_send_congestion_window` | long | Minimum tcp send congestion window size. This field is only available if flowLogsEnableTcpStats is enabled | | `tcp_mean_smooth_rtt` | long | Mean smooth RTT in micro seconds. This field is only available if flowLogsEnableTcpStats is enabled | | `tcp_max_smooth_rtt` | long | Maximum smooth RTT in micro seconds. This field is only available if flowLogsEnableTcpStats is enabled | | `tcp_mean_min_rtt` | long | Mean MinRTT in micro seconds. This field is only available if flowLogsEnableTcpStats is enabled | | `tcp_max_min_rtt` | long | Maximum MinRTT in micro seconds. This field is only available if flowLogsEnableTcpStats is enabled | | `tcp_mean_mss` | long | Mean TCP MSS. This field is only available if flowLogsEnableTcpStats is enabled | | `tcp_min_mss` | long | Minimum TCP MSS. This field is only available if flowLogsEnableTcpStats is enabled | | `tcp_total_retransmissions` | long | Total retransmitted packets. This field is only available if flowLogsEnableTcpStats is enabled | | `tcp_lost_packets` | long | Total lost packets. This field is only available if flowLogsEnableTcpStats is enabled | | `tcp_unrecovered_to` | long | Total unrecovered timeouts. This field is only available if flowLogsEnableTcpStats is enabled | ### Format of the policies field[​](#format-of-the-policies-field) The `policies` field contains four sub-fields, `all_policies`, `enforced_policies`, `pending_policies`, and `transit_policies`. | Name | Datatype | Description | | ------------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `all_policies` | array of keywords | **Deprecated.** This field is a legacy field calculated at flow start time that combines both enforced and staged policies. Because it mixes actual verdicts with simulated ones, it can be misleading and is planned for removal. **Note:** This field may still appear in logs from older clusters reporting to a newer management cluster. | | `enforced_policies` | array of keywords | The policies that were actually enforced on the flow. This list is determined by the dataplane when the flow starts and remains fixed for the flow's lifetime. It represents the concrete actions taken on the traffic. | | `pending_policies` | array of keywords | A simulation of what policy evaluation would look like at the time of log generation. It represents a hypothetical restart of the flow where all staged policies are considered active. This field captures types of updates such as: (1) changes to active policies made after the flow started (for example, if a new policy is added that would deny the flow if it restarted), (2) staged policies treated as if they were enforced, and (3) domain-based policies evaluated based on the DNS state at the time of log generation. | | `transit_policies` | array of keywords | Policies applied to traffic transiting through the node (for example, policies applied to host endpoints with `applyOnForward: true`) or during pre-DNAT processing. This field functions like `enforced_policies` but specifically for traffic being evaluated at the host endpoint. **Note**: This field is only populated when `flowLogsPolicyScope` is set to `AllPolicies` in the `FelixConfiguration`. | Each entry in the list has the following format: ```text |||| ``` Where, - `` numbers the order in which the rules were hit, starting with `0`. > **SUCCESS:** Sort the entries of the list by the `` to see the order that rules were hit. The entries are displayed in random order due to the way they are stored in the datastore. - `` is the name of the policy tier containing the policy, or `__PROFILE__` for a rule derived from a `Profile` resource (this is the internal datatype used to represent a Kubernetes namespace and its associated "default allow" rule). - `` is the name of the policy/profile; its format depends on the type of policy: - `.` for Calico Cloud `GlobalNetworkPolicy`. - `/knp.default.` for Kubernetes `NetworkPolicy`. - `/.` for Calico Cloud `NetworkPolicy`. Staged policy names are prefixed with "staged:". - `` is the action performed by the rule; one of `allow`, `deny`, `pass`. - `` if non-negative, is the index of the rule that was matched within the policy, starting with 0. Otherwise, a special value: - `-1` means the reporting endpoint was selected by the policy but no rule matched. The traffic hit the default action for the tier. In this case, the `` is selected arbitrarily from the set of policies within the tier that apply to the endpoint. - `-2` means "unknown". The rule index was not recorded. ### Flow log example, with `no aggregation`[​](#flow-log-example-with-no-aggregation) A flow log with aggregation level 0, `no aggregation`, might look like: ```text { "start_time": 1597166083, "end_time": 1597166383, "source_ip": "192.168.47.9", "source_name": "access-6b687c8dcb-zn5s2", "source_name_aggr": "access-6b687c8dcb-*", "source_namespace": "policy-demo", "source_port": 42106, "source_type": "wep", "source_labels": { "labels": [ "pod-template-hash=6b687c8dcb", "app=access" ] }, "dest_ip": "192.168.138.79", "dest_name": "nginx-86c57db685-h6792", "dest_name_aggr": "nginx-86c57db685-*", "dest_namespace": "policy-demo", "dest_port": 80, "dest_type": "wep", "dest_labels": { "labels": [ "pod-template-hash=86c57db685", "app=nginx" ] }, "proto": "tcp", "action": "allow", "reporter": "dst", "policies": { "all_policies": [ "0|tier1|ns1/tier1.policy1|pass|3", "1|tier2|ns2/staged:tier1.policy1|deny|1", "2|default|policy-demo/default.access-nginx|allow|2" ], "enforced_policies": [ "0|tier1|ns1/tier1.policy1|pass|3", "1|default|policy-demo/default.access-nginx|allow|2" ], "pending_policies": [ "0|tier1|ns1/tier1.policy1|pass|3", "1|tier2|ns2/staged:tier1.policy1|deny|1", ] }, "bytes_in": 388, "bytes_out": 1113, "num_flows": 1, "num_flows_started": 1, "num_flows_completed": 1, "packets_in": 6, "packets_out": 5, "http_requests_allowed_in": 0, "http_requests_denied_in": 0, "original_source_ips": null, "num_original_source_ips": 0, "host": "bz-n8kf-kadm-node-1", "@timestamp": 1597166383000 } ``` The log shows an incoming connection reported by the destination node, allowed by a policy on port 80. The **`start_time`** and **`end_time`** describe the aggregation period (5 min.) During this interval, one flow (**`"num_flow": 1`**) was recorded. At higher aggregation levels, flows from endpoints performing the same operation and originating from the same Deployment/ReplicaSet are grouped into a single log. In this example, the common source endpoints that are prefixed with **`access-6b687c8dcb-`**. Parameters like **`source_ip`** may be dropped and set to **`null`**, depending on the aggregation level. As aggregation levels increase, more flows will be grouped together based on your data. For more details on aggregation levels, see [configure flow log aggregation](https://docs.tigera.io/calico-cloud/observability/elastic/flow/aggregation). ### Filter flow logs ## Big picture[​](#big-picture) Filter Calico Cloud flow logs. ## Value[​](#value) Filter Calico Cloud flow logs to suppress logs of low significance, and troubleshoot threats. ## Concepts[​](#concepts) ### Container monitoring tools versus flow logs[​](#container-monitoring-tools-versus-flow-logs) Container monitoring tools are good for monitoring Kubernetes and orchestrated workloads for CPU usage, network usage, and log aggregation. For example, a data monitoring tool can tell if a pod has turned into a bitcoin miner based on it using more than normal CPU. Calico Cloud flow logs provide continuous records of every single packet sent/received by all pods in your Kubernetes cluster. Note that flow logs do not contain all packet data; only the number of packets/bytes that were sent between specific IP/ports, and when. In the previous monitoring tool example, Calico Cloud flow logs could see the packets running to/from the bitcoin mining network. Calico Cloud flow logs tell you when a pod is compromised, specifically: - Where a pod is sending data to - If the pod is talking to a known command-and-control server - Other pods that the compromised pod has been talking to (so you can see if they're compromised too) ### Flow log format[​](#flow-log-format) A flow log contains these space-delimited fields (unless filtered out). ```text startTime endTime srcType srcNamespace srcName srcLabels dstType dstNamespace dstName dstLabels srcIP dstIP proto srcPort dstPort numFlows numFlowsStarted numFlowsCompleted reporter packetsIn packetsOut bytesIn bytesOut action ``` **Example** ```text 1528842551 1528842851 wep dev rails-81531* - wep dev memcached-38456* - - - 6 - 3000 7 3 4 out 154 61 70111 49404 allow ``` - Fields that are not enabled or are aggregated, are noted by `-` - Aggregated names (such as “pod prefix”), are noted by `*` at the end of the name - If `srcName` or `dstName` fields contain only a `*`, aggregation was performed using other means (such as specific labels), and no unique prefix was present. ## How to[​](#how-to) - [Create flow log filters](#create-flow-log-filters) - [Add filters to ConfigMap file](#add-filters-to-configmap-file) ### Create flow log filters[​](#create-flow-log-filters) Create your [fluentd filters](https://docs.fluentd.org/filter/grep). **Example: filter out a specific namespace** This example filters out all flow logs whose source or destination namespace is "dev". Additional namespaces could be filtered by adjusting the regular expression "pattern"s, or by adding additional `exclude` blocks. ```text @type grep key source_namespace pattern dev key dest_namespace pattern dev ``` **Example: filter out internet traffic to a specific deployment** This example filters inbound internet traffic to the deployment with pods named, `nginx-internet-*`. Note the use of the `and` directive to filter out traffic that is both to the deployment, and from the internet (source `pub`). ```text @type grep key dest_name_aggr pattern ^nginx-internet key source_name_aggr pattern pub ``` ### Add filters to ConfigMap file[​](#add-filters-to-configmap-file) 1. Create a `filters` directory with a file called `flow` with your desired filters. If you are also adding [dns filters](https://docs.tigera.io/calico-cloud/observability/elastic/dns/filtering-dns), add the `dns` file to the directory. 2. Create the `fluentd-filters` ConfigMap in the `tigera-operator` namespace with the following command. ```bash kubectl create configmap fluentd-filters -n tigera-operator --from-file=filters ``` ## Additional resources[​](#additional-resources) - [Flow log aggregation](https://docs.tigera.io/calico-cloud/observability/elastic/flow/aggregation) - [Archive logs to storage](https://docs.tigera.io/calico-cloud/observability/elastic/archive-storage) ### Configure flow log aggregation ## Big picture[​](#big-picture) Configure flow log aggregation level to reduce log volume and costs. ## Value[​](#value) Beyond using filtering to suppress flow logs, Calico Cloud provides controls to aggregate flow logs. Although aggressive aggregation levels reduce flow volume and costs, it can also reduce visibility into specific metadata of allowed and denied traffic. Review this article to see which level of aggregation is suitable for your implementation. ## Concepts[​](#concepts) ### Volume and cost versus visibility[​](#volume-and-cost-versus-visibility) Calico Cloud enables flow log aggregation by default using default aggregation levels that balance log volume with comprehensive visibility. The defaults assume that most users do not need to see pod IP information (due to the ephemeral nature of pod IP address allocation) for allowed traffic, but provides additional details on denied traffic that is more likely to need investigation. However, it all depends on your deployment; we recommend reviewing aggregation levels to understand what information gets grouped (and thus suppressed from view). ### Aggregation levels[​](#aggregation-levels) For allowed flows, the default aggregation level is 2, and for denied flows the default aggregation level is 1. #### Level 0: no aggregation[​](#level-0-no-aggregation) Create separate flow logs for each distinct 5-tuple (protocol, source and destination IP and port) observed. This level is not recommended due to high log volumes. #### Level 1: aggregate source ports[​](#level-1-aggregate-source-ports) Aggregate flow data relating to multiple connections that only differ in source port. This reduces log volume by discarding source port information, which is usually ephemeral and of little value. #### Level 2: aggregate IPs and source ports[​](#level-2-aggregate-ips-and-source-ports) In addition to the above, aggregate flows that have related sources and destinations into a single log depending on the source and destination type. - Pods created by the same pod controller (Deployments, ReplicaSets, etc.) are combined and identified by their common pod prefix, - IP addresses in the same NetworkSet are aggregated under that shared NetworkSet, - When no more precise identity is known, arbitrary IP addresses are aggregated to either public (`pub`) and private (`pvt`) as defined in RFC1918. #### Level 3: aggregate IPs, source and dest ports[​](#level-3-aggregate-ips-source-and-dest-ports) In addition to the above, aggregate flows with different destination ports that otherwise share together. This is intended to reduce log volumes in situations where a lot of network probing is expected (for example by `nmap`) but is not typically needed unless log analysis indicates it will be beneficial. ### Understanding aggregation level differences[​](#understanding-aggregation-level-differences) Here are examples of pod-to-pod flows, highlighting the differences between flow logs at various aggregation levels. By suppressing the source port, aggregation level 1 minimizes the flow logs generated for applications that make many connections to the same destination. The two flows originating from `client-a` without aggregation are combined into one. In Kubernetes, pod controllers (Deployments, ReplicaSets, etc.) automatically name the pods they create with a common prefix. For example, the pods `nginx-1` and `nginx-2` are created by the ReplicaSet `nginx`. At aggregation level 2 that prefix is used to aggregate flow log entries and is indicated with an asterisk (`*`) at the end of the name. The flows originating from `client-a` and `client-b` are combined into a single flow log. Finally, level 3 combines flows between matching pod prefixes that target distinct destination ports, as seen in the last example row below. Aggregation is currently performed on a per-node basis, but this behavior may change. Certain other fields are always kept separate and not aggregated together such as `action` (i.e. denied traffic and allowed traffic will always be logged separately). | | | **Src Traffic** | | | **Dst Traffic** | | | **Packet counts** | | | ------------------------ | --------- | --------------- | ------- | -------- | --------------- | ------- | -------- | ----------------- | ----------- | | **Aggr lvl** | **Flows** | **Name** | **IP** | **Port** | **Name** | **IP** | **Port** | **Pkt in** | **Pkt out** | | 0 (no aggregation) | 4 | client-a | 1.1.1.1 | 45556 | nginx-1 | 2.2.2.2 | 80 | 1 | 2 | | | | client-b | 1.1.2.2 | 45666 | nginx-2 | 2.2.3.3 | 80 | 2 | 2 | | | | client-a | 1.1.1.1 | 65533 | nginx-1 | 2.2.2.2 | 80 | 1 | 3 | | | | client-c | 1.1.3.3 | 65534 | nginx-2 | 2.2.3.3 | 8080 | 3 | 4 | | 1 (src port) | 3 | client-a | 1.1.1.1 | - | nginx-1 | 2.2.2.2 | 80 | 2 | 5 | | | | client-b | 1.1.2.2 | - | nginx-1 | 2.2.2.2 | 80 | 2 | 2 | | | | client-c | 1.1.3.3 | - | nginx-2 | 2.2.3.3 | 8080 | 3 | 4 | | 2 (+src/dest pod-prefix) | 2 | client-\* | - | - | nginx-\* | - | 80 | 4 | 7 | | | | client-\* | - | - | nginx-\* | - | 8080 | 3 | 4 | | 3 (+dest port) | 1 | client-\* | - | - | nginx-\* | - | - | 7 | 11 | ## How to[​](#how-to) - [Verify existing aggregation level](#verify-existing-aggregation-level) - [Change default aggregation level](#change-default-aggregation-level) - [Troubleshoot logs with aggregation levels](#troubleshoot-logs-with-aggregation-levels) ### Verify existing aggregation level[​](#verify-existing-aggregation-level) Use the following command: ```bash kubectl get felixconfiguration -o yaml ``` ### Change default aggregation level[​](#change-default-aggregation-level) Before [changing the default aggregation level](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig#aggregationkind), note the following: - Although any change in aggregation level affects flow log volume, lowering the aggregation number (especially to `0` for no aggregation) will cause significant impacts to log storage. If you allow more flow logs, ensure that you provision more log storage. - Verify that the parameters that you want to see in your aggregation level are not already [filtered](https://docs.tigera.io/calico-cloud/observability/elastic/flow/filtering). ### Troubleshoot logs with aggregation levels[​](#troubleshoot-logs-with-aggregation-levels) When you use flow log aggregation, sometimes you may see two Alerts, ![two-alerts](https://docs.tigera.io/assets/images/two-alerts-f0590a647ed819689ba2197056fae708.png) along with two flow log entries. Note that the entries are identical except for the slight timestamp difference. ![two-logs](https://docs.tigera.io/assets/images/two-logs-3281b556ed4a5aba08df6239134f2858.png) The reason you may see two entries is because of the interaction between the aggregation interval, and the time interval to export logs (`flowLogsFlushInterval`). In each aggregation interval, connections/connection attempts can be started or completed. However, flow logs do not start/stop when a connection starts/stops. Let’s assume the default export logs “flush” time of 10 seconds. If a connection is started in one flush interval, but terminates in the next, it is recorded across two entries. To get visibility into flow logs to differentiate the entries, go to Service Graph, flow logs tab, and look at these fields: `num_flows`, `num_flows_started`, and `num_flows_completed`. The underlying reason for this overlap is a dependency on Linux conntrack, which provides the lifetime of stats that Calico Cloud tracks across different protocols (TCP, ICMP, UDP). For example, for UDP and ICMP, Calico Cloud waits for a conntrack entry to timeout before it considers a “connection” closed, and this is usually greater than 10 seconds. ## Additional resources[​](#additional-resources) - [Archive logs to storage](https://docs.tigera.io/calico-cloud/observability/elastic/archive-storage) ### Enable HostEndpoint reporting in flow logs ## Big picture[​](#big-picture) Enable Calico Cloud flow logs to report HostEndpoint information. ## Value[​](#value) Get visibility into the network activity at the HostEndpoint level using Calico Cloud flow logs. ## Before you begin[​](#before-you-begin) **Limitations** - HostEndpoint reporting is only supported on Kubernetes nodes. - Flow logs on ApplyOnForward policies are currently not supported. As a result, a policy blocking traffic at the host level from forwarding to a workload endpoint would not result in a flow log from the host endpoint. ## How to[​](#how-to) ### Enable HostEndpoint reporting[​](#enable-hostendpoint-reporting) To enable reporting HostEndpoint metadata in flow logs, use the following command: ```text kubectl patch felixconfiguration default -p '{"spec":{"flowLogsEnableHostEndpoint":true}}' ``` ## Additional resources[​](#additional-resources) - [Protect Kubernetes nodes](https://docs.tigera.io/calico-cloud/network-policy/hosts/kubernetes-nodes) ### Enabling TCP socket stats in flow logs ## Big picture[​](#big-picture) Configure Calico Cloud to collect additional TCP socket statistics. While this feature is available in both iptables and eBPF data plane modes, it uses eBPF to collect the statistics. Therefore it requires a recent Linux kernel (at least v5.3.0/v4.18.0-193 for RHEL). ## Value[​](#value) Get visibility into the network activity at the socket level using Calico Cloud flow logs. ## Concepts[​](#concepts) ### eBPF TC programs[​](#ebpf-tc-programs) eBPF is a Linux kernel technology that allows safe mini-programs to be attached to various hooks inside the kernel. This feature leverages eBPF to look up the TCP socket associated with packets flowing through an interface and sends them to userspace for addition to flow logs. ## Before you begin[​](#before-you-begin) Ensure that your kernel contains support for eBPF that Calico Cloud uses. The minimum supported kernel for tcp socket stats is: `v5.3.0`. For distros based on RHEL, the minimum kernel version is `v4.18.0-193`. # How to ### Enable tcp stats collection[​](#enable-tcp-stats-collection) Calico Cloud can be configured to enable tcp socket stats collection on supported Linux kernels using the command: ```text kubectl patch felixconfiguration default -p '{"spec":{"flowLogsCollectTcpStats":true}}' ``` ### View tcp stats in flow logs using Kibana.[​](#view-tcp-stats-in-flow-logs-using-kibana) Navigate to the Kibana Flow logs dashboard to view tcp stats associated with a flow log entry. The additional fields collected are `tcp_mean_send_congestion_window`, `tcp_min_send_congestion_window`, `tcp_mean_smooth_rtt`, `tcp_max_smooth_rtt`, `tcp_mean_min_rtt`, `tcp_max_min_rtt`, `tcp_mean_mss`, `tcp_min_mss`, `tcp_total_retransmissions`, `tcp_lost_packets`, `tcp_unrecovered_to`. Information about these fields are described in the [Flow log datatype document](https://docs.tigera.io/calico-cloud/observability/elastic/flow/datatypes) ### Enable process-level information in flow logs ## Big picture[​](#big-picture) Configure Calico Cloud to collect process executable path and arguments and add them to flow logs. ## Value[​](#value) Get visibility into the network activity at the process level using Calico Cloud flow logs. ## Concepts[​](#concepts) ### eBPF kprobe programs[​](#ebpf-kprobe-programs) eBPF is a Linux kernel technology that allows safe mini-programs to be attached to various hooks inside the kernel. To collect the path and arguments of short-lived processes, this feature uses an eBPF kprobe program. ### Host's PID namespace[​](#hosts-pid-namespace) For long-lived processes, path and arguments are read from `/proc/pid/cmdline`. This requires access to the host's PID namespace. If the access is not available then the process path and arguments will only be captured (by the eBPF kprobes) for newly-created processes. ## Before you begin[​](#before-you-begin) Ensure that your kernel contains support for eBPF kprobes that Calico Cloud uses. The minimum supported kernel for this is feature is: `v4.4.0`. ## Privileges[​](#privileges) For full functionality, this feature requires the `calico-node` `DaemonSet` to have access to the host's PID namespace. The Tigera Operator will automatically grant this extra privilege to the daemonset if the feature is enabled in the operator's LogCollector resource, as described below. # How to ### Enable process path and argument collection[​](#enable-process-path-and-argument-collection) Calico Cloud can be configured to enable process path and argument collection on supported Linux kernels using the command: ```text kubectl patch logcollector.operator.tigera.io tigera-secure --type merge -p '{"spec":{"collectProcessPath":"Enabled"}}' ``` Enabling/Disabling collectProcessPath causes a rolling update of the `calico-node DaemonSet`. ### View process path and arguments in flow logs using Kibana[​](#view-process-path-and-arguments-in-flow-logs-using-kibana) Navigate to the Kibana Flow logs dashboard to view process path and arguments associated with a flow log entry. The executable path will appear in the `process_name` field and `process_args` will have the executable arguments. Executable path and arguments cannot be collected under certain circumstances, in that `process_name` will have the task name and `process_args` will be empty. Information about these fields are described in the [Flow log datatype document](https://docs.tigera.io/calico-cloud/observability/elastic/flow/datatypes) ### Audit logs ## Big picture[​](#big-picture) Calico Cloud audit logs provide security teams and auditors historical data of all changes to resources over time. ## Concepts[​](#concepts) ### Resources used in audit logs[​](#resources-used-in-audit-logs) Calico Cloud audit logs are enabled by default for the following resources: - Global networkpolicies - Network policies - Staged global networkpolicies - Staged networkpolicies - Staged Kubernetes network policies - Global network sets - Network sets - Tiers - Host endpoints ### Audit logs in the web console[​](#audit-logs-in-the-web-console) Calico Cloud audit logs are displayed in the Timeline dashboard in the web console. You can filter logs, and export data in .json or .yaml formats. ![audit-logs](https://docs.tigera.io/assets/images/audit-logs-7b3d418ce8b1b523012e7a79d531e024.png) Audit logs are also visible in the Kibana dashboard (indexed by, `tigera_secure_ee_audit_ee`), and are useful for looking at policy differences. ![kibana-auditlogs](https://docs.tigera.io/assets/images/kibana-auditlogs-efb62f9638ccc4e044e59daba91ddbac.png) Finally, audit logs provide the core data for compliance reports. ![compliance-reports](https://docs.tigera.io/assets/images/configuration-compliance-5b72e093c1f763b1ccfbb42113656b08.png) ## Required next step[​](#required-next-step) **Kubernetes resources** are also used in compliance reports and other audit-related features, but they are not enabled by default. You must enable Kubernetes resources through the Kubernetes API server. If you miss this step, some compliance reports will not work, and audit trails will not provide a complete view to your security team. - [Enable Kubernetes audit logs](https://docs.tigera.io/calico-cloud/observability/kube-audit) ### Manage DNS logs for Calico Cloud ## [📄️ Query DNS logs](https://docs.tigera.io/calico-cloud/observability/elastic/dns/dns-logs) [Key/value pairs of DNS activity logs and how to construct queries.](https://docs.tigera.io/calico-cloud/observability/elastic/dns/dns-logs) ## [📄️ Filter DNS logs](https://docs.tigera.io/calico-cloud/observability/elastic/dns/filtering-dns) [Suppress DNS logs of low significance using filters.](https://docs.tigera.io/calico-cloud/observability/elastic/dns/filtering-dns) ### Query DNS logs Calico Cloud pushes DNS activity logs to Elasticsearch, for DNS information that is obtained from [trusted DNS servers](https://docs.tigera.io/calico-cloud/network-policy/domain-based-policy#trusted-dns-servers). The following table details the key/value pairs in the JSON blob, including their [Elasticsearch datatype](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html). This information should assist you in constructing queries. | Name | Datatype | Description | | ------------------ | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `start_time` | date | When the collection of the log began in UNIX timestamp format. | | `end_time` | date | When the collection of the log concluded in UNIX timestamp format. | | `type` | keyword | This field contains one of the following values: ● `LOG`: Indicates that this is a normal DNS activity log. ● `UNLOGGED`: Indicates that this log is reporting DNS activity that could not be logged in detail because of [DNSLogsFilePerNodeLimit](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig#spec). | | `count` | long | When `type` is: ● `LOG`: How many DNS lookups there were, during the log collection interval, with details matching this log. ● `UNLOGGED`: The number of DNS responses that could not be logged in detail because of [DNSLogsFilePerNodeLimit](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig#spec). In this case none of the following fields are provided. | | `client_ip` | ip | The IP address of the client pod. A null value indicates aggregation. | | `client_name` | keyword | This field contains one of the following values: ● The name of the client pod. ● `-`: the name of the pod was aggregated. Check `client_name_aggr` for the pod name prefix. | | `client_name_aggr` | keyword | The aggregated name of the client pod. | | `client_namespace` | keyword | Namespace of the client pod. | | `client_labels` | array of keywords | Labels applied to the client pod. With aggregation, the label name/value pairs that are common to all aggregated clients. | | `qname` | keyword | The domain name that was looked up. | | `qtype` | keyword | The type of the DNS query (e.g. A, AAAA). | | `qclass` | keyword | The class of the DNS query (e.g. IN). | | `rcode` | keyword | The result code of the DNS query response (e.g. NoError, NXDomain). | | `rrsets` | nested | Detailed DNS query response data - see below. | | `servers` | nested | Details of the DNS servers that provided this response. | | `latency_count` | long | The number of lookups for which latency was measured. (The same as `count` above, unless some DNS requests were missed, or latency reporting is disabled - see `dnsLogsLatency` in the [FelixConfiguration resource](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig).) | | `latency_mean` | long | Mean latency, in nanoseconds. | | `latency_max` | long | Max latency, in nanoseconds. | Each nested `rrsets` object contains response data for a particular name and a particular type and class of response information. Its key/value pairs are as follows. | Name | Datatype | Description | | ------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------- | | `name` | keyword | The domain name that this information is for. | | `type` | keyword | The type of the information (e.g. A, AAAA). | | `class` | keyword | The class of the information (e.g. IN). | | `rdata` | array of keywords | Array of data, for the name, of that type and class. For example, when `type` is A, this is an array of IPs for `name`. | Each nested `servers` object provides details of a DNS server that provided the information in the containing log. Its key/value pairs are as follows. | Name | Datatype | Description | | ----------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ip` | ip | The IP address of the DNS server. | | `name` | keyword | This field contains one of the following values: ● The name of the DNS server pod. ● `-`: the DNS server is not a pod. | | `name_aggr` | keyword | This field contains one of the following values: ● The aggregated name of the DNS server pod. ● `pvt`: the DNS server is not a pod. Its IP address belongs to a private subnet. ● `pub`: the DNS server is not a pod. Its IP address does not belong to a private subnet. It is probably on the public internet. | | `namespace` | keyword | Namespace of the DNS server pod, or `-` if the DNS server is not a pod. | | `labels` | array of keywords | Labels applied to the DNS server pod or host endpoint; empty if there are no labels or the DNS server is not a pod or host endpoint. | The `latency_*` fields provide information about the latency of the DNS lookups that contributed to this log. For each successful DNS lookup Calico Cloud measures the time between when the DNS request was sent and when the corresponding DNS response was received. ## Query DNS log fields[​](#query-dns-log-fields) After a set of DNS logs has accumulated in Elasticsearch, you can perform many interesting queries. For example, if you query on: - `qname`, you can find all of the DNS response information that was provided to clients trying to resolve a particular domain name - `rrsets.rdata`, you can find all of the DNS lookups that included a particular IP address in their response data. ### Filter DNS logs Calico Cloud supports filtering out DNS logs based on user provided configuration. Use filtering to suppress logs of low significance. ## Configure DNS filtering[​](#configure-dns-filtering) DNS log filtering is configured through a ConfigMap in the `tigera-operator` namespace. To enable DNS log filtering, follow these steps: 1. Create a `filters` directory with a file named `dns` with the contents of your desired filter using [Filter configuration files](#filter-configuration-files). If you are also adding [flow filters](https://docs.tigera.io/calico-cloud/observability/elastic/flow/filtering) also add the `flow` file to the directory. 2. Create the `fluentd-filters` ConfigMap in the `tigera-operator` namespace with the following command. ```bash kubectl create configmap fluentd-filters -n tigera-operator --from-file=filters ``` ## Filter configuration files[​](#filter-configuration-files) The filters defined by the ConfigMap are inserted into the fluentd configuration file. The [upstream fluentd documentation](https://docs.fluentd.org/filter/grep) describes how to write fluentd filters. The [DNS log schema](https://docs.tigera.io/calico-cloud/observability/elastic/dns/dns-logs) can be referred to for the specification of the various fields you can filter based on. Remember to ensure that the config file is properly indented in the ConfigMap. ## Example 1: filter out cluster-internal lookups[​](#example-1-filter-out-cluster-internal-lookups) This example filters out lookups for domain names ending with ".cluster.local". More logs could be filtered by adjusting the regular expression "pattern", or by adding additional `exclude` blocks. ```text @type grep key qname pattern /\.cluster\.local$/ ``` ## Example 2: keep logs only for particular domain names[​](#example-2-keep-logs-only-for-particular-domain-names) This example will filter out all logs *except* those for domain names ending `.co.uk`. ```text @type grep key qname pattern /\.co\.uk$/ ``` ### BGP logs Calico Cloud pushes BGP activity logs to Elasticsearch. To view them, go to the Discovery view, and from the dropdown menu, select `tigera_secure_ee_bgp.*` to view the collected BIRD and BIRD6 logs. The following table details key/value pairs for constructing queries, including their Elasticsearch datatype. | Name | Datatype | Description | | ------------ | -------- | ------------------------------------------------------------------------------------------------------ | | `logtime` | date | When the log was collected in UTC timestamp format. | | `host` | keyword | The name of the node where log was collected. | | `ip_version` | keyword | Contains one of the following values: ● `IPv4`: Log from BIRD process ● `IPv6`: Log from BIRD6 process | | `message` | text | The message contained in the log. | Once a set of BGP logs has accumulated in Elasticsearch, you can perform many interesting queries. Depending on the field that you want to query, different techniques are required. For example: - To view BGP logs only for IPv4 or IPv6, query on the `ip_version` field and sort by `logtime` - To see all logs from a specific node, query on the `host` field - To view events in the cluster, query on the `message` field ### L7 logs ## [📄️ Configure L7 logs](https://docs.tigera.io/calico-cloud/observability/elastic/l7/configure) [Configure and aggregate L7 logs.](https://docs.tigera.io/calico-cloud/observability/elastic/l7/configure) ## [📄️ L7 log data types](https://docs.tigera.io/calico-cloud/observability/elastic/l7/datatypes) [L7 data that Calico Cloud sends to Elasticsearch.](https://docs.tigera.io/calico-cloud/observability/elastic/l7/datatypes) ### Configure L7 logs ## Big picture[​](#big-picture) Deploy Envoy and use Calico Cloud L7 logs to monitor application activity. ## Value[​](#value) Just like L3/4 Calico Cloud logs, platform operators and development teams want visibility into L7 logs to see how applications are interacting with each other. Calico Cloud flow logs only display which workloads are communicating with each other, not the specific request details. Calico Cloud provides visibility into L7 traffic without the need for a service mesh. L7 logs are also key for detecting anomalous behaviors like attempts to access applications, restricted URLs, and scans for particular URLs. ## Concepts[​](#concepts) ### About L7 logs[​](#about-l7-logs) L7 logs capture application interactions from HTTP header data in requests. Data shows what is actually sent in communications between specific pods, providing more specificity than flow logs. (Flow logs capture data only from connections for workload interactions). Calico Cloud collects L7 logs by sending the selected traffic through an Envoy proxy. L7 logs are visible in the web console, service graph, in the HTTP tab. ## Before you begin[​](#before-you-begin) **Not supported** - GKE **Limitations** - L7 log collection is not supported for host-networked client pods. - When selecting and deselecting traffic for L7 log collection, active connections may be disrupted. ## How to[​](#how-to) - [Configure Felix for log data collection](#configure-felix-for-log-data-collection) - [Configure L7 logs](#configure-l7-logs) - [View L7 logs in the web console](#view-l7-logs-in-manager-ui) ### Configure Felix for log data collection[​](#configure-felix-for-log-data-collection) 1. Configure L7 log aggregation, retention, and reporting. For help, see [Felix Configuration documentation](https://docs.tigera.io/calico-cloud/reference/component-resources/node/felix/configuration#calico-enterprise-specific-configuration). ### Configure L7 logs[​](#configure-l7-logs-1) In this step, you will configure L7 logs, select logs for collection, and test the configuration. **Configure the ApplicationLayer resource for L7 logs** 1. Create or update the [ApplicationLayer](https://docs.tigera.io/calico-cloud/reference/installation/api#applicationlayer) resource named, `tigera-secure`. Example: ```bash kubectl apply -f - < -n projectcalico.org/l7-logging=true ``` 2. To disable the L7 log collection, remove the annotation. ```bash kubectl annotate svc -n projectcalico.org/l7-logging- ``` After annotating a service for L7 log collection, only newly-established connections through that service are proxied by Envoy. Connections established before the service is annotated are not proxied or interrupted, and no logs are generated. Conversely, when a service is deselected, any previous connections established through the annotated service continue to be proxied by Envoy until they are terminated, and logs are generated. **Test your configuration** 1. Identify the path to access your cluster. Where `` can be: - Public address of your cluster/service or - Cluster IP of your application's service (if testing within the cluster) 2. `curl` your service with a command similar to the following. You will see `Server` header as `envoy`. ```bash curl --head :/ ``` ### View L7 logs in the web console[​](#view-l7-logs-in-the-web-console) **Service Graph** To view L7 logs in Service Graph: 1. In the web console left navbar, click **Service Graph**. 2. In the bottom pane you will see L7 logs in the HTTP tab. ![l7-logs](https://docs.tigera.io/assets/images/l7-logs-cad7cc9111742ad666c84646eb7523f4.png) **Kibana** To view L7 logs by index pattern in Kibana: 1. In the web console left navbar, click **Kibana**. 2. In the new Kibana browser, click the hamburger icon in the top left corner, and select **Analytics**, **Discover**. 3. Select the index pattern, `tigera_secure_ee_l7`. ### L7 log data types ## Big picture[​](#big-picture) Calico Cloud sends the following data to Elasticsearch. The following table details the key/value pairs in the JSON blob, including their [Elasticsearch datatype](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html). | Name | Datatype | Description | | ------------------------ | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `host` | keyword | Name of the node that collected the L7 log entry. | | `start_time` | date | Start time of log collection in UNIX timestamp format. | | `end_time` | date | End time of log collection in UNIX timestamp format. | | `bytes_in` | long | Number of incoming bytes since the last export. | | `bytes_out` | long | Number of outgoing bytes since the last export. | | `duration_mean` | long | Mean duration time of all the requests that match this combination of L7 data in nanoseconds. | | `duration_max` | long | Max duration time of all the requests that match this combination of L7 data in nanoseconds. | | `count` | long | Number of requests that match this combination of L7 data. | | `src_name_aggr` | keyword | Contains one of the following values: - Aggregated name of the source pod. - `pvt`: endpoint is not a pod. Its IP address belongs to a private subnet. - `pub`: endpoint is not a pod. Its IP address does not belong to a private subnet. It is probably an endpoint on the public internet. | | `src_namespace` | keyword | Namespace of the source endpoint. | | `src_type` | keyword | Source endpoint type. Possible values: - `wep`: A workload endpoint, a pod in Kubernetes. - `ns`: A Networkset. If multiple Networksets match, then the one with the longest prefix match is chosen. - `net`: A Network. The IP address did not fall into a known endpoint type. | | `dest_name_aggr` | keyword | Contains one of the following values: - Aggregated name of the destination pod. - `pvt`: endpoint is not a pod. Its IP address belongs to a private subnet. - `pub`: endpoint is not a pod. Its IP address does not belong to a private subnet. It is probably an endpoint on the public internet. | | `dest_namespace` | keyword | Namespace of the destination endpoint. | | `dest_type` | keyword | Destination endpoint type. Possible values: - `wep`: A workload endpoint, a pod in Kubernetes. - `ns`: A Networkset. If multiple Networksets match, then the one with the longest prefix match is chosen. - `net`: A Network. The IP address did not fall into a known endpoint type. | | `dest_service_name` | keyword | Name of the destination service. This may be empty if the request was not made against a service. | | `dest_service_namespace` | keyword | Namespace of the destination service. This may be empty if the request was not made against a service. | | `dest_service_port` | long | Destination service port. | | `url` | keyword | URL that the request was made against. | | `response_code` | keyword | Response code returned by the request. | | `method` | keyword | HTTP method for the request. | | `user_agent` | keyword | User agent of the request. | | `type` | keyword | Type of request made. Possible values include `tcp`, `tls`, and `html/`. | ### Kubernetes audit logs ## Big picture[​](#big-picture) Enable Kubernetes audit logs so security teams and auditors can see all the changes to Kubernetes resources over time. ## Concepts[​](#concepts) ### About Kubernetes audit logs[​](#about-kubernetes-audit-logs) **Kubernetes resources** are used in Calico Cloud compliance reports and other audit-related features, but they are not enabled by default. You must enable Kubernetes resources through the Kubernetes API server. If you miss this step, some compliance reports will not work, and audit trails will not provide a complete view to your security team. You must enable the following Kubernetes resources for each cluster: - Pod - Namespace - Service account - Network policy - Endpoints ### Audit logs in the web console[​](#audit-logs-in-the-web-console) Like Calico Cloud audit logs, Kubernetes audit logs are displayed in the web console in the Timeline dashboard, Kibana dashboard (indexed by, `tigera_secure_ee_audit_kube`), and provide the core data for compliance reports. ## Before you begin[​](#before-you-begin) **Unsupported** - AKS - GKE - OpenShift - TKG ## How to[​](#how-to) Enable Kubernetes audit logs in the Kubernetes API server: **Tab: Kubernetes on-premises** ### Enable audit logs for Kubernetes resources[​](#enable-audit-logs-for-kubernetes-resources) At a minimum, enable audit logs for these resources that are involved in network policy: - Pod - Namespace - ServiceAccount - NetworkPolicy (Kubernetes/OpenShift) - Endpoints **Sample policy** The following sample policy audits changes to Kubernetes Pod, Namespace, ServiceAccount, Endpoints and NetworkPolicy resources. To add other audit logs for resources beyond network policy, see the [Kubernetes docs](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/), or review this function for inspiration (which generates the GKE audit policy). ```yaml apiVersion: audit.k8s.io/v1beta1 kind: Policy omitStages: - RequestReceived rules: - level: RequestResponse verbs: - create - patch - update - delete resources: - group: networking.k8s.io resources: ['networkpolicies'] - group: extensions resources: ['networkpolicies'] - group: '' resources: ['pods', 'namespaces', 'serviceaccounts', 'endpoints'] ``` ### Enable Kubernetes audit logs for Calico Cloud[​](#enable-kubernetes-audit-logs-for-calico-cloud) The following updates require a restart to the Kubernetes API Server. To enable Kubernetes resource audit logs to be read by Calico Cloud in fluentd, follow these steps. On the Kubernetes API Server, update these flags. - `--audit-log-path=/var/log/calico/audit/kube-audit.log` - `--audit-policy-file=` For help with flags, see kube-apiserver flags. For help with audit logging, see Kubernetes audit logging documentation. Distribute the audit policy file to all control plane nodes, ensuring that it is available to the Kubernetes API server (e.g. by volume mounting it into the pods). Restart the Kubernetes API server. The restart command depends on how you installed Kubernetes. **Tab: EKS** ### Enable audit logs for Kubernetes resources[​](#enable-audit-logs-for-kubernetes-resources-1) At a minimum, enable audit logs for these resources that are involved in network policy: - Pod - Namespace - ServiceAccount - NetworkPolicy (Kubernetes/OpenShift) - Endpoints **Sample policy** The following sample policy audits changes to Kubernetes Pod, Namespace, ServiceAccount, Endpoints and NetworkPolicy resources. To add other audit logs for resources beyond network policy, see the [Kubernetes docs](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/), or review this function for inspiration (which generates the GKE audit policy). ```yaml apiVersion: audit.k8s.io/v1beta1 kind: Policy omitStages: - RequestReceived rules: - level: RequestResponse verbs: - create - patch - update - delete resources: - group: networking.k8s.io resources: ['networkpolicies'] - group: extensions resources: ['networkpolicies'] - group: '' resources: ['pods', 'namespaces', 'serviceaccounts', 'endpoints'] ``` ### Enable audit logs in EKS[​](#enable-audit-logs-in-eks) Amazon EKS writes Kubernetes audit logs to [Amazon Cloudwatch logs](https://aws.amazon.com/cloudwatch/). 1. In the EKS management console, access your EKS cluster. 2. Under **Logging**, click **Update**. 3. Enable the **Audit** option, and click **Update**. ![Audit Log](https://docs.tigera.io/img/calico-enterprise/audit-log.png) 4. Wait for the update to complete. The blue progress bar at the top of the page displays the message, “Cluster config update in progress.” 5. Under **Logging**, **Cloudwatch**, make a note of the URL value for a later step, then click the link. ![Cloudwatch Logging](https://docs.tigera.io/img/calico-enterprise/cloudwatch-url.png) 6. In the **Log Streams** list, make note of the common prefix (for example, kube-apiserver-audit) for a later step. ![Log Streams](https://docs.tigera.io/img/calico-enterprise/log-streams.png) 7. Make note of the region where the cluster is hosted (for example, `us-west-2`) for a later step. ### Create a restricted AWS user for compliance reporting[​](#create-a-restricted-aws-user-for-compliance-reporting) 1. Go to the **AWS IAM console** and add a user. 2. On the **Add user** page, make these changes: a. Select **Access type**, **Programmatic access**. ![Programmatic access](https://docs.tigera.io/img/calico-enterprise/programmatic-access.png) b. In the Set permissions section, select the policy, `CloudWatchLogsReadOnlyAccess` to set read only permissions. ![Cloudwatch URL](https://docs.tigera.io/img/calico-enterprise/cloudwatch-readonly.png) 3. Optional: In the **Add tags** section, add a tag for the user based on your cluster information. 4. Click **Submit** to create a restricted user. ### Update Calico Cloud log collector with EKS values[​](#update-calico-cloud-log-collector-with-eks-values) 1. Update the `tigera-secure` LogCollector resource with values from the EKS configuration. where: - `additionalSources`: Section where EKS Cloudwatch logs are specified. - `eksCloudwatchLog`: Configuration section containing EKS Cloudwatch logs. - `fetchInterval`: Interval in seconds for Calico Cloud to get logs from Cloudwatch. Default: 60 seconds, this fetches 1MB every 60 seconds, adjust it based number on CRUD operations performed on cluster resource. - `groupName`: Name of the `Log Group` (value from "Enable audit logs in EKS") - `region`: AWS region where EKS cluster is hosted (value from "Enable audit logs in EKS") - `streamPrefix`: Prefix of `Log Stream` (value from "Enable audit logs in EKS") **Example** ```yaml apiVersion: operator.tigera.io/v1 kind: LogCollector metadata: name: tigera-secure spec: additionalSources: eksCloudwatchLog: fetchInterval: 60 groupName: /aws/eks/mitch-eks-kube-audit-log-forwarder/cluster region: us-west-2 streamPrefix: kube-apiserver-audit- status: state: Ready ``` ### Configure authentication between Calico Cloud and Cloudwatch logs[​](#configure-authentication-between-calico-cloud-and-cloudwatch-logs) In this step, you add AWS authentication information to enable Calico Cloud to get logs from the EKS Cloudwatch instance. Add a Secret with the name, `tigera-eks-log-forwarder-secret` in the namespace, `tigera-operator`, and the AWS [Security Credentials](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html) in the data section. ```yaml apiVersion: v1 kind: Secret metadata: name: tigera-eks-log-forwarder-secret namespace: tigera-operator type: Opaque data: aws-id: $(echo -n | base64 -w0) aws-key: $(echo -n | base64 -w0) ``` **Tab: AWS using kOps** ### Enable audit logs for Kubernetes resources[​](#enable-audit-logs-for-kubernetes-resources-2) At a minimum, enable audit logs for these resources that are involved in network policy: - Pod - Namespace - ServiceAccount - NetworkPolicy (Kubernetes/OpenShift) - Endpoints **Sample policy** The following sample policy audits changes to Kubernetes Pod, Namespace, ServiceAccount, Endpoints and NetworkPolicy resources. To add other audit logs for resources beyond network policy, see the [Kubernetes docs](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/), or review this function for inspiration (which generates the GKE audit policy). ```yaml apiVersion: audit.k8s.io/v1beta1 kind: Policy omitStages: - RequestReceived rules: - level: RequestResponse verbs: - create - patch - update - delete resources: - group: networking.k8s.io resources: ['networkpolicies'] - group: extensions resources: ['networkpolicies'] - group: '' resources: ['pods', 'namespaces', 'serviceaccounts', 'endpoints'] ``` Follow these instructions to enable audit logs for [AWS using kOps](https://kops.sigs.k8s.io/cluster_spec/#audit-logging). Note that `auditLogPath` should be `/var/log/calico/audit/kube-audit.log`. ### iptables logs ## About iptables logs[​](#about-iptables-logs) iptables logs are produced by [policy audit mode](#policy-audit-mode) or by using the `Log` action in either [Network Policy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) or [Global Network Policy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy). These logs are written to syslog (specifically the `/dev/log` socket) on the nodes where the events are generated. Collection, rotation and other management of these logs is provided by your syslog agent, for example, journald or rsyslogd. ## Policy audit mode[​](#policy-audit-mode) Calico Cloud adds a Felix option `DropActionOverride` that configures how the `deny` `action` in a [Rule](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy#rule) is interpreted. It can add logs for denied packets, or even allow the traffic through. See the [Felix configuration reference](https://docs.tigera.io/calico-cloud/reference/component-resources/node/felix/configuration#calico-cloud-specific-configuration) for information on how to configure this option. `DropActionOverride` controls what happens to each packet that is denied by the current Calico Cloud policy, i.e., by the ordered combination of all the configured policies and profiles that apply to that packet. It may be set to one of the following values: - `Drop` - `Accept` - `LogAndDrop` - `LogAndAccept` Normally the `Drop` or `LogAndDrop` value should be used, as dropping a packet is the obvious implication of that packet being denied. However when experimenting, or debugging a scenario that is not behaving as you expect, the "Accept" and "LogAndAccept" values can be useful: then the packet will be still be allowed through. When one of the `LogAnd*` values is set, each denied packet is logged in syslog, with an entry like this: ```text May 18 18:42:44 ubuntu kernel: [ 1156.246182] calico-drop: IN=tunl0 OUT=cali76be879f658 MAC= SRC=192.168.128.30 DST=192.168.157.26 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=56743 DF PROTO=TCP SPT=56248 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0 MARK=0xa000000 ``` Note that [Denied Packet Metrics](https://docs.tigera.io/calico-cloud/operations/monitor/metrics/) are independent of the `DropActionOverride` setting. Specifically, if packets that would normally be denied are being allowed through by a setting of `Accept` or `LogAndAccept`, those packets still contribute to the denied packet metrics as normal. For example, to set a `DropActionOverride` for `myhost` to log then drop denied packets: Edit the FelixConfiguration object for the `myhost` Node. ```bash kubectl patch felixconfiguration.p node.myhost --type='merge' -p \ '{"spec":{"dropActionOverride":"LogAndDrop"}}' ``` For a global setting, modify the `default` FelixConfiguration resource. --- ## Threat defense ### Threat defense Use real-time monitoring to detect and block threats to your cluster. ##### [Security event management](https://docs.tigera.io/calico-cloud/threat/security-event-management) [Manage security events from your cluster in a single place.](https://docs.tigera.io/calico-cloud/threat/security-event-management) ##### [Container threat detection](https://docs.tigera.io/calico-cloud/threat/container-threat-detection) [Threat detection for containerized workloads.](https://docs.tigera.io/calico-cloud/threat/container-threat-detection) ##### [Trace and alert on suspicious domains](https://docs.tigera.io/calico-cloud/threat/suspicious-domains) [Add threat intelligence feeds to trace DNS queries that involve suspicious domains.](https://docs.tigera.io/calico-cloud/threat/suspicious-domains) ##### [Trace and block suspicious IPs](https://docs.tigera.io/calico-cloud/threat/suspicious-ips) [Add threat intelligence feeds to trace network flows of suspicious IP addresses, and optionally block traffic to them.](https://docs.tigera.io/calico-cloud/threat/suspicious-ips) ##### [Workload-based Web Application Firewall (WAF)](https://docs.tigera.io/calico-cloud/threat/web-application-firewall) [Configure Calico to use with Layer 7 Web Application Firewall.](https://docs.tigera.io/calico-cloud/threat/web-application-firewall) ##### [Webhooks for security events](https://docs.tigera.io/calico-cloud/threat/configuring-webhooks) [Use webhooks to send security event alerts to third-party systems.](https://docs.tigera.io/calico-cloud/threat/configuring-webhooks) ##### [Deploy a web application firewall with Calico Ingress Gateway](https://docs.tigera.io/calico-cloud/threat/deploying-waf-ingress-gateway) [Deploy WAF with ingress gateways](https://docs.tigera.io/calico-cloud/threat/deploying-waf-ingress-gateway) ##### [Deep packet inspection](https://docs.tigera.io/calico-cloud/threat/deeppacketinspection) [Monitor live traffic for malicious activities.](https://docs.tigera.io/calico-cloud/threat/deeppacketinspection) ##### [Anonymization attacks](https://docs.tigera.io/calico-cloud/threat/tor-vpn-feed-and-dashboard) [Detect and analyze malicious anonymization activity using Tor-VPN feeds.](https://docs.tigera.io/calico-cloud/threat/tor-vpn-feed-and-dashboard) ### Webhooks for security event alerts You can configure Calico Cloud webhooks to post security alerts directly to Slack, Jira, Alertmanager, or any custom HTTP endpoint. ## Before you begin[​](#before-you-begin) Your target application must be configured to receive data from the Calico Cloud webhook. - **Slack.** You must have a webhook URL for the Slack app that you want Calico Cloud to send alerts to. See [Sending messages using Incoming Webhooks](https://api.slack.com/messaging/webhooks) for more information. - **Jira.** You must have an API token for an Atlassian user account that has write permissions to your Jira instance. See [Manage API tokens for your Atlassian account](https://support.atlassian.com/atlassian-account/docs/manage-api-tokens-for-your-atlassian-account/) for details on how to obtain an API token. You also need: - Your Atlassian site URL. If you access Jira at the URL `https://.atlassian.net/jira`, then your site URL is `.atlassian.net`. - A Jira project key. This is the Jira project where your Calico Cloud webhook creates new issues. This user associated with your API token must have write permissions to this project. - **Alertmanager.** You must have a webhook URL for your Alertmanager instance to send alerts to. The webhook URL for an Alertmanager service accessible from within your cluster is: ```bash http://..svc.cluster.local:9093/api/v2/alerts ``` Replace the following: - ``: the name of your Alertmanager service. - ``: the namespace associated with your Alertmanager service. If no namespace is specified in the Alertmanager service resource, then this value is `default`. Alertmanager integration in Calico Cloud leverages the API v2 exposed by the Alertmanager. For more information, see the [Alertmanager API documentation](https://prometheus.io/docs/alerting/latest/alerts_api/). - **Generic JSON.** You must have a webhook URL for any other application you want the Calico Cloud webhook to send alerts to. ## Create a webhook for security event alerts[​](#create-a-webhook-for-security-event-alerts) 1. In the web console, select **Activity** > **Webhooks**, and then click **Create your first webhook**. 2. Enter a **Name** for your webhook, select which **Event types** you want to get alerts for, and, under **Type**, select whether to configure the webhook for Slack, Jira, Alertmanager, or for generic JSON output. 3. Complete the fields for your webhook type and click **Create Webhook**. 4. To check that your webhook is working, find the webhook on the **Webhooks** page and click **Actions > Test Webhook**. If your webhook is set up correctly, you should generate a HTTP response. Example: Successful HTTP response ```bash 200 OK Webhook sent successfully ``` ### Security event management Manage security events from your cluster in a single place. ## Value[​](#value) Security events indicate that a threat actor may be present in your Kubernetes cluster. For example, a DNS request to a malicious hostname, a triggered WAF rule, or the opening of a sensitive file. Calico Cloud provides security engineers and incident response teams with a single dashboard to manage threat alerts. Benefits include: - A filtered list of critical events with recommended remediation - Identify impacts on applications - Understand the scope and frequency of the issue - Manage alert noise by dismissing events (show/hide) - Manage alert noise by creating exceptions ## Before you begin[​](#before-you-begin) **Required** - [WAF is enabled](https://docs.tigera.io/calico-cloud/threat/web-application-firewall) **Limitations** - Only WAF basic security events. Over time, the dashboard will contain a full range of Calico Cloud security events. - You cannot control which users can view or edit the page using fine-grained role-based access controls ## Security Events Dashboard[​](#security-events-dashboard) The **Security Events Dashboard** page gives you a high-level view of recent security events. You can use this visual reference to get an overall sense of your cluster's health. If you find anything that merits further investigation, you can click on an event for more details. - In the web console, go to **Threat defense > Security Events Dashboard**. ![Security Events Dashboard](https://docs.tigera.io/assets/images/security-events-dashboard-a6374a59a0b732fd6a5adbb1d46a9566.png) ## Security Events[​](#security-events) The **Security Events** page lists all the security events that have been detected for your cluster. You can view and filter your security events to focus on - In the web console, go to **Threat Defense > Security Events**. ### Dismiss a security event[​](#dismiss-a-security-event) You can clear your security events list by dismissing events that you've finished reviewing. When you dismiss an event, that event is no longer visibile in the list. 1. In the web console, go to **Threat Defense > Security Events**. 2. Find a security event in the list, and then click **Action > Dismiss Security Event**. ### Create a security event exception[​](#create-a-security-event-exception) You can prevent certain kinds of security events from appearing in the list by creating a security event exception. This is helpful if you want to reduce alert noise for workloads that you know are safe. When you create an exception, all matching security events are removed from the security events list. Future matches will not appear in the list. 1. In the web console, go to **Threat Defense > Security Events**. 2. Find a security event in the list, and then click **Action > Add exception**. 3. On the **Create an Exception** dialog, select a scope for the exception and click **Create Exception**. You can manage your exceptions by clicking **Threat Defense > Security Events > Exceptions**. You can browse, edit, and delete exceptions on the list. ### UI help[​](#ui-help) **Event details page** Provides actions to remediate the detection and stop the attack from progressing. For example: ![waf-security](https://docs.tigera.io/assets/images/waf-security-events-latest-95179eb74e4f8c189de4122f1320e1fb.png) **Severity** Calico Cloud calculates severity (Critical, High, Medium, Low) using a combination of NIST CVSS 3.0 and MITRE IDs. **MITRE IDs** Multiple MITRE IDs may be associated with a security event. **Attack Vector** - Network - Process - File **MITRE Tactic** (based on the [MITRE tactics](https://attack.mitre.org/tactics/enterprise/)) includes a specific path, method, or scenario that can compromise cluster security. Valid entries: | Tactic | Target | Attack techniques | | ---------------------------------------------------------------- | ------------------------------- | ---------------------------------------------------------------------------------------------------- | | [Initial access](https://attack.mitre.org/tactics/TA0001/) | Network | Gain an initial foothold within a network using various entry vectors. | | [Execution](https://attack.mitre.org/tactics/TA0002/) | Code in local or remove systems | Control code running on local or remote systems using malicious code. | | [Impact](https://attack.mitre.org/tactics/TA0040/) | Systems and data | Disrupt availability or compromise integrity by manipulating business and operational processes. | | [Persistence](https://attack.mitre.org/tactics/TA0003/) | Maintain footholds | Maintain access to systems across restarts, credential changes, and other interruptions. | | [Privilege Escalation](https://attack.mitre.org/tactics/TA0004/) | Access permissions | Access higher-level permissions on a system or network. | | [Defense Evasion](https://attack.mitre.org/tactics/TA0005/) | Avoid detection | Masquerade and hide malware to avoid detection to compromise software, data, scripts, and processes. | | [Discovery](https://attack.mitre.org/tactics/TA0007/) | Determine your environment | Gain knowledge about your system and internal network. | ### Frequently asked questions[​](#frequently-asked-questions) **How is the recommended remediation determined?** The Tigera Security Research team maps MITRE IDs to events and provides the recommended remediation. **Will I see all Calico Cloud alerts in this dashboard?** No. Calico Cloud security events do not encompass all types of alerts nor all security alert types; they only contain alerts for threats. Alerts for vulnerabilities detected in a container image, or misconfigurations in your Kubernetes cluster, are displayed in their respective dashboards. However, when vulnerabilities or misconfigurations are exploited by an attacker, those indicators of an attack are considered security events. **What does dismissing a security event do?** Dismissing a security event hides it from view. **Why are some fields in columns blank?** Security events generated from older managed clusters will not have values for the new fields (for example, MITRE IDs). You can dismiss these events. **Where can I view security event logs?** Go to: **Logs**, Kibana index, `tigera_secure_ee_events`. ### Trace and block suspicious IPs ## Big picture[​](#big-picture) Add threat intelligence feeds to Calico Cloud to trace network flows of suspicious IP addresses, and optionally block traffic to suspicious IPs. ## Value[​](#value) Calico Cloud integrates with threat intelligence feeds so you can detect when your Kubernetes clusters communicate with suspicious IPs. When communications are detected, an anomaly detection dashboard in the UI shows the full context, including which pod(s) were involved so you can analyze and remediate. You can also use a threat intelligence feed to power a dynamic deny-list, either to or from a specific group of sensitive pods, or your entire cluster. ## Concepts[​](#concepts) Calico Cloud supports pull methods for updating threat feeds. Use this method for fully automated threat feed updates without user intervention. ### Suspicious IPs: test before you block[​](#suspicious-ips-test-before-you-block) There are many different types of threat intelligence feeds (community-curated, company-paid, and internally-developed) that you can choose to monitor in Calico Cloud. We recommend that you assess the threat feed contents for false positives before blocking based on the feed. If you decide to block, test a subset of your workloads before rolling to production to ensure legitimate application traffic is not blocked. ## Before you begin...[​](#before-you-begin) ### Required[​](#required) Privileges to manage GlobalThreatFeed and GlobalNetworkPolicy. ### Recommended[​](#recommended) We recommend that you turn down the aggregation of flow logs sent to Elasticsearch for configuring threat feeds. If you do not adjust flow logs, Calico Cloud aggregates over the external IPs for allowed traffic, and threat feed searches will not provide useful results (unless the traffic is denied by policy). Go to: [FelixConfiguration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig) and set the field, **flowLogsFileAggregationKindForAllowed** to **1**. You can adjust the flow logs by running the following command: ```bash kubectl patch felixconfiguration default --type='merge' -p '{"spec":{"flowLogsFileAggregationKindForAllowed":1}}' ``` ## How to[​](#how-to) This section describes how to pull threat feeds to Calico Cloud, and block traffic to a cluster for a suspicious IP. - [Pull threat feed updates](#pull-threat-feed-updates) - [Block traffic to a cluster](#block-traffic-to-a-cluster) ### Pull threat feed updates[​](#pull-threat-feed-updates) To add threat feeds to Calico Cloud for automatic updates (default is once a day), the threat feed(s) must be available using HTTP(S), and return a newline-separated list of IP addresses or prefixes in CIDR notation. #### Using the web console[​](#using-the-web-console) 1. From the web console, select **Threat Feeds** --> **Add Feed**. 2. Add your threat feed on the Add a New Threat Feed window. For example: - **Feed Name**: feodo-tracker - **Description**: This is the feodo-tracker threat feed. - **URL**: [https://feodotracker.abuse.ch/downloads/ipblocklist.txt](https://feodotracker.abuse.ch/downloads/ipblocklist.txt) - **Content type**: IPSet - **Labels**: Choose a label from the list. 3. Click **Save Changes**. From the **Action** menu, you can view or edit the details that you entered and can download the manifest file. > Go to the Security Events page to view events that are generated when an IP is displayed on the threat feed list. For more information, see [Manage alerts](https://docs.tigera.io/calico-cloud/observability/alerts). When you create a global threat feed in the web console, network traffic is not automatically blocked. If you find suspicious IPs on the Security Events page, you need to create a network policy to block the traffic. For help with policy, see [Block traffic to a cluster](#block-traffic-to-a-cluster). #### Using CLIs[​](#using-clis) 1. Create the GlobalThreatFeed YAML and save it to file. The simplest example of this looks like the following. Replace the **name** and the **URL** with your feed. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalThreatFeed metadata: name: my-threat-feed spec: content: IPSet mode: Enabled description: 'This is my threat feed' feedType: Custom pull: http: url: https://my.threatfeed.com/deny-list ``` 2. Add the global threat feed to the cluster. ```bash kubectl apply -f ``` > Go to the Security Events page to view events that are generated when an IP is displayed on the threat feed list. For more information, see [Manage alerts](https://docs.tigera.io/calico-cloud/observability/alerts). ### Block traffic to a cluster[​](#block-traffic-to-a-cluster) Create a new/edit existing threat feed to include the globalNetworkSet stanza, setting the labels you want to use to represent the deny-listed IPs. This stanza instructs Calico Cloud to search for flows to and from the listed IP addresses, and maintain a GlobalNetworkSet containing the IP addresses. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalThreatFeed metadata: name: sample-global-threat-feed spec: content: IPSet mode: Enabled description: 'This is the sample global threat feed' feedType: Custom pull: http: url: https://an.example.threat.feed/deny-list globalNetworkSet: labels: security-action: block ``` 1. Add the global threat feed to the cluster. ```bash kubectl apply -f ``` 2. Create a GlobalNetworkPolicy that blocks traffic based on the threat feed, by selecting sources or destinations using the labels you assigned in step 1. For example, the following GlobalNetworkPolicy blocks all traffic coming into the cluster if it came from any of the suspicious IPs. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: default.blockthreats spec: tier: default selector: all() types: - Ingress ingress: - action: Deny source: selector: security-action == 'block' ``` 3. Add the global network policy to the cluster. ```bash kubectl apply -f ``` ## Tutorial[​](#tutorial) In this tutorial, we’ll walk through setting up a threat feed to search for connections to suspicious IPs. Then, we’ll use the same threat feed to block traffic to those IPs. We will use the free [FEODO botnet tracker](https://feodotracker.abuse.ch/) from abuse.ch that lists IP addresses associated with command and control servers. But the steps are the same for your commercial or internal threat feeds. If you haven’t already adjusted your [aggregation flows](#before-you-begin), we recommend it before you start. ### Configure the threat feed[​](#configure-the-threat-feed) 1. Create a file called feodo-tracker.yaml with the following contents: ```yaml apiVersion: projectcalico.org/v3 kind: GlobalThreatFeed metadata: name: feodo-tracker spec: content: IPSet mode: Enabled description: 'This is the feodo-tracker threat feed' feedType: Custom pull: http: url: https://feodotracker.abuse.ch/downloads/ipblocklist.txt ``` This pulls updates using the default period of once per day. See the [Global Resource Threat Feed API](https://docs.tigera.io/calico-cloud/reference/resources/globalthreatfeed) for all configuration options. 2. Add the feed to your cluster. ```bash kubectl apply -f feodo-tracker.yaml ``` ### Check search results[​](#check-search-results) Open the Calico Cloud web console, and navigate to the “Security Events” page. If any of your pods have been communicating with the IP addresses in the FEODO tracker feed, you will see the results listed on this page. It is normal to not see any events listed on this page. ### Block pods from contacting IPs[​](#block-pods-from-contacting-ips) If you have high confidence in the IP addresses listed as malicious in a threat feed, you can take stronger action than just searching for connections after the fact. For example, the FEODO tracker lists IP addresses used by command and control servers for botnets. We can configure Calico Cloud to block all egress traffic to addresses on this list. It is strongly recommended that you assess the contents of a threat feed for false positives before using it as a deny-list, and that you apply it to a test subset of your workloads before rolling out application or cluster-wide. Failure to do so could cause legitimate application traffic to be blocked and could lead to an outage in your application. In this demo, we will apply the policy only to a test workload (so we do not impact other traffic). 1. Create a file called **tf-ubuntu.yaml** with the following contents: ```yaml apiVersion: v1 kind: Pod metadata: labels: docs.tigera.io-tutorial: threat-feed name: tf-ubuntu spec: nodeSelector: kubernetes.io/os: linux containers: - command: - sleep - '3600' image: ubuntu name: test ``` 2. Apply the pod configuration. ```bash kubectl apply -f tf-ubuntu.yaml ``` 3. Edit the feodo-tracker.yaml to include a globalNetworkSet stanza: ```yaml apiVersion: projectcalico.org/v3 kind: GlobalThreatFeed metadata: name: feodo-tracker spec: content: IPSet mode: Enabled description: 'This is the feodo-tracker threat feed' feedType: Custom pull: http: url: https://feodotracker.abuse.ch/downloads/ipblocklist.txt globalNetworkSet: labels: docs.tigera.io-threat-feed: feodo ``` 4. Reapply the new YAML. ```bash kubectl apply -f feodo-tracker.yaml ``` 5. Verify that the GlobalNetworkSet is created. ```bash kubectl get globalnetworksets threatfeed.feodo-tracker -o yaml ``` ### Apply global network policy[​](#apply-global-network-policy) We will now apply a GlobalNetworkPolicy that blocks the test workload from connecting to any IPs in the threat feed. 1. Create a file called block-feodo.yaml with the following contents: ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: default.block-feodo spec: tier: default selector: docs.tigera.io-tutorial == 'threat-feed' types: - Egress egress: - action: Deny destination: selector: docs.tigera.io-threat-feed == 'feodo' - action: Allow ``` 2. Apply this policy to the cluster ```bash kubectl apply -f block-feodo.yaml ``` ### Verify policy on test workload[​](#verify-policy-on-test-workload) We will verify the policy from the test workload that we created earlier. 1. Get a shell in the pod by running the following command: ```bash kubectl exec -it tf-ubuntu -- bash ``` You should get a prompt inside the pod. 2. Install the ping command. ```bash apt update && apt install iputils-ping ``` 3. Ping a known safe IP (like 8.8.8.8, Google’s public DNS server). ```bash ping 8.8.8.8 ``` 4. Open the [FEODO tracker list](https://feodotracker.abuse.ch/downloads/ipblocklist.txt) and choose an IP on the list to ping. You should not get connectivity, and the pings will show up as denied traffic in the flow logs. ## Additional resources[​](#additional-resources) See [GlobalThreatFeed](https://docs.tigera.io/calico-cloud/reference/resources/globalthreatfeed) resource definition for all configuration options. ### Trace and alert on suspicious domains ## Big picture[​](#big-picture) Add threat intelligence feeds to Calico Cloud to trace DNS queries involving suspicious domains. ## Value[​](#value) Calico Cloud integrates with threat intelligence feeds so you can detect when endpoints in your Kubernetes clusters query DNS for suspicious domains, or receive answers with suspicious domains. When events are detected, an anomaly detection dashboard in the UI shows the full context, including which pod(s) were involved so you can analyze and remediate. ## Concepts[​](#concepts) Calico Cloud supports pull methods for updating threat feeds. Use this method for fully automated threat feed updates without user intervention. ### Domain name threat feeds[​](#domain-name-threat-feeds) A best practice is to develop an allow-list of "known-good" domains that particular applications or services must access, and then [enforce this allow-list with network policy](https://docs.tigera.io/calico-cloud/network-policy/domain-based-policy). In addition to allow-lists, you can use threat feeds to monitor your cluster for DNS queries to known malicious or suspicious domain names. Calico Cloud monitors DNS queries and generates alerts for any that are listed in your threat feed. Threat feeds for domain names associated with malicious **egress** activity (e.g. command and control (C2) servers or data exfiltration), provide the most security value. Threat feeds that associate domain names with malicious **ingress** activity (e.g. port scans or IP sweeps) are less useful since these activities do not cause endpoints in your cluster to query DNS. It is better to consider [IP-based threat feeds](https://docs.tigera.io/calico-cloud/threat/suspicious-ips) for ingress activity. ## Before you begin...[​](#before-you-begin) ### Required[​](#required) Privileges to manage GlobalThreatFeed. ### Recommended[​](#recommended) We recommend that you turn down the aggregation of DNS logs sent to Elasticsearch for configuring threat feeds. If you do not adjust DNS log aggregation settings, Calico Cloud aggregates DNS queries from workloads in the same replica set. This means if a suspicious DNS query is detected, you will only know which replica set made the query and not which specific pod. Go to: [FelixConfiguration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig) and set the field, **dnsLogsFileAggregationKind** to **0** to log individual pods separately. ## How to[​](#how-to) This section describes how to pull threat feeds to Calico Cloud. ### Pull threat feed updates[​](#pull-threat-feed-updates) To add threat feeds to Calico Cloud for automatic updates (default is once a day), the threat feed(s) must be available using HTTP(S), and return a newline-separated list of domain names. #### Using the web console[​](#using-the-web-console) 1. From the web console, select **Threat Feeds** --> **Add Feed**. 2. Add your threat feed on the Add a New Threat Feed window. For example: - **Feed Name**: feodo-tracker - **Description**: This is my threat feed based on domains. - **URL**: [https://my.threatfeed.com/deny-list](https://my.threatfeed.com/deny-list) - **Content type**: DomainNameSet - **Labels**: Choose a label from the list. 3. Click **Save Changes**. From the **Action** menu, you can view or edit the details that you entered and can download the manifest file. > Go to the Security Events page to view events that are generated when an endpoint in the cluster queries a name on the list. For more information, see [Manage alerts](https://docs.tigera.io/calico-cloud/observability/alerts). #### Using CLIs[​](#using-clis) 1. Create the GlobalThreatFeed YAML and save it to file. The simplest example of this looks like the following. Replace the **name** and the **URL** with your feed. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalThreatFeed metadata: name: my-threat-feed spec: content: DomainNameSet mode: Enabled description: 'This is my threat feed' feedType: Custom pull: http: url: https://my.threatfeed.com/deny-list ``` 2. Add the global threat feed to the cluster. ```shell kubectl apply -f ``` > Go to the Security Events page to view events that are generated when an endpoint in the cluster queries a name on the list. For more information, see [Manage alerts](https://docs.tigera.io/calico-cloud/observability/alerts). ## Additional resources[​](#additional-resources) See [GlobalThreatFeed](https://docs.tigera.io/calico-cloud/reference/resources/globalthreatfeed) resource definition for all configuration options. ### Anonymization attacks ## Big picture[​](#big-picture) Detect and analyze malicious anonymization activity using Tor-VPN feeds. ## Value[​](#value) **Tor and VPN infrastructure** are used in enabling anonymous communication, where an attacker can leverage anonymity to scan, attack or compromise the target. It’s hard for network security teams to track malicious actors using such anonymization tools. Hence **Tor and VPN feeds** come into play where the feeds track all the Tor bulk exit nodes as well as most of the anonymizing VPN infrastructure on the internet. **The Tor-VPN Dashboard** helps network security teams to monitor and respond to any detected activity where they have a clusterwide view and granular control over logs which is critical in stopping the possible attack in early stages. ## Concepts[​](#concepts) ### About Tor and VPN threats[​](#about-tor-and-vpn-threats) **Tor** is a popular anonymization network on the internet. It is also popular among the malicious actors, hacktivist groups, criminal enterprises as the infrastructure hides the real identity of an attacker carrying out malicious activities. To track down such attackers, Tor historically was subject to investigation by various state level intelligence agencies from US and UK for criminal activities such as Silk Road marketplace, Mirai Botnet C\&C. Though it’s not possible to completely de-anonymize the attacker. Hence **Tor bulk exit feed** came into existence to track all the Tor exit IPs over the internet to know attackers using the Tor infrastructure. Over the years, many Tor flaws became public and attackers evolved to leverage Tor network with additional VPN layers. There are many individual VPN providers which have the anonymizing infrastructure. Attackers can use these new breed of VPN providers with existing options like Tor to make sure of anonymity. To help security teams, the **X4B vpn feed** detects all the major VPN providers on the internet. ### Tor-VPN feed types[​](#tor-vpn-feed-types) **Tor Bulk Exit feed** The Tor Bulk Exit feed lists available Tor exit nodes on the internet which are used by Tor network. The list is continuously updated and maintained by the Tor project. An attacker using Tor network, is likely to use one of the bulk exit nodes to connect to your infrastructure. The network security teams can detect such activity with Tor bulk exit feed and investigate as required. **X4B VPN feed** In recent times it became a trend to use multiple anonymization networks to hide real attacker identity. There are lots of lists of open proxies and tor nodes on the web, but surprisingly few usable ones dedicated to VPN providers and datacenters. This list combines known VPN netblocks and ASNs owned by datacenters and VPN providers. This list doesn't list all VPNs, but should list the vast majority of common ones. ### The Calico Cloud Tor-VPN dashboard[​](#the-calico-cloud-tor-vpn-dashboard) The Tor-VPN dashboard helps network security teams to monitor and respond to any detected activity by Tor and VPN feeds. It provides a cluster context to the detection and shows multiple artifacts e.g. flow logs, filtering controls, a tag cloud and line graph to analyze the activity and respond faster. The Tor-VPN dashboard may be accessed as below: - Log in to the Calico Cloud web console, and go to **kibana**, select **dashboard**, and select **Tor-VPN Dashboard**. ## Before you begin...[​](#before-you-begin) ### Required[​](#required) Privileges to manage GlobalThreatFeed i.e. clusterrole `intrusion-detection-controller` ### Recommended[​](#recommended) We recommend that you turn down the aggregation of flow logs sent to Elasticsearch for configuring threat feeds. If you do not adjust flow logs, Calico Cloud aggregates over the external IP addresses for allowed traffic, and threat feed searches will not provide useful results (unless the traffic is denied by policy). Go to: [FelixConfiguration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig) and set the field, **flowLogsFileAggregationKindForAllowed** to **1**. ## How to[​](#how-to) In this section we will look at how to add Tor and VPN feeds to Calico Cloud. Installation process is straightforward as below. 1. Add threat feed to the cluster. For VPN Feed, ```shell kubectl apply -f https://downloads.tigera.io/ee/v3.22.0-2.0/manifests/threatdef/vpn-feed.yaml ``` For Tor Bulk Exit Feed, ```shell kubectl apply -f https://downloads.tigera.io/ee/v3.22.0-2.0/manifests/threatdef/tor-exit-feed.yaml ``` 2. Now, you can monitor the Dashboard for any malicious activity. The dashboard can be found at the Calico Cloud web console, go to "kibana" and then go to "Dashboard". Select "Tor-VPN Dashboard". 3. Additionally, feeds can be checked using following command: ```shell kubectl get globalthreatfeeds ``` ## Additional resources[​](#additional-resources) - See [GlobalThreatFeed](https://docs.tigera.io/calico-cloud/reference/resources/globalthreatfeed) resource definition for all configuration options. - Check example to Trace and block Suspicious IPs [Here](https://docs.tigera.io/calico-cloud/threat/suspicious-ips) ### Deep packet inspection ## Big picture[​](#big-picture) Configure deep packet inspection (DPI) in clusters to get alerts on compromised resources. ## Value[​](#value) Security teams need to run DPI quickly in response to unusual network traffic in clusters so they can identify potential threats. Also, it is critical to run DPI on select workloads (not all) to efficiently make use of cluster resources and minimize the impact of false positives. Calico Cloud provides an easy way to perform DPI using [Snort community rules](https://www.snort.org/downloads/#rule-downloads). You can disable DPI at any time, selectively configure for namespaces and endpoints, and alerts are generated in the Alerts dashboard in the web console. ## Concepts[​](#concepts) For each deep packet inspection resource (DeepPacketInspection), Calico Cloud creates a live network monitor that inspects the header and payload information of packets that match the Snort community rules. Whenever malicious activities are suspected, an alert is automatically added to the Alerts page in the Calico Cloud web console. Calico Cloud DPI uses AF\_PACKET, a Linux socket that allows an application to receive and send raw packets. It is commonly used for troubleshooting (like tcpdump and Wireshark), but also for network intrusion detection. For details, see [AF\_Packet](https://man7.org/linux/man-pages/man7/packet.7.html). ## Before you begin[​](#before-you-begin) **Not supported:** - Multi-nic setup - Calico Cloud nodes running Windows hosts ## How to[​](#how-to) - [Configure deep packet inspection](#configure-deep-packet-inspection) - [Configure resource requirements](#configure-resource-requirements) - [Access alerts](#access-alerts) - [Verify deep packet inspection is running](#verify-deep-packet-inspection-is-running) ### Configure deep packet inspection[​](#configure-deep-packet-inspection) Create a YAML file containing one or more [DeepPacketInspection](https://docs.tigera.io/calico-cloud/reference/resources/deeppacketinspection) resources and apply it to your cluster. ```bash kubectl apply -f ``` To stop deep packet inspection, delete the DeepPacketInspection resource from your cluster. ```bash kubectl delete -f ``` **Examples of selecting workloads** Following is a basic example that selects a single workload that has the label `k8s-app` with the value `nginx`. ```yaml apiVersion: projectcalico.org/v3 kind: DeepPacketInspection metadata: name: sample-dpi-nginx namespace: sample spec: selector: k8s-app == "nginx" ``` In the following example, we select all workload endpoints in the `sample` namespace. ```yaml apiVersion: projectcalico.org/v3 kind: DeepPacketInspection metadata: name: sample-dpi-all namespace: sample spec: selector: all() ``` ### Configure resource requirements[​](#configure-resource-requirements) Adjust the CPU and RAM used for performing deep packet inspection by updating the [component resource in IntrusionDetection](https://docs.tigera.io/calico-cloud/reference/installation/api#intrusiondetectioncomponentresource). For a data transfer rate of 1GB/sec on workload endpoints being monitored, we recommend a minimum of 1 CPU and 1GB RAM. The following example configures deep packet inspection to use a maximum of 1 CPU and 1GB RAM. ```yaml apiVersion: operator.tigera.io/v1 kind: IntrusionDetection metadata: name: tigera-secure spec: componentResources: - componentName: DeepPacketInspection resourceRequirements: limits: cpu: '1' memory: 1Gi requests: cpu: 100m memory: 100Mi ``` ### Access alerts[​](#access-alerts) The alerts generated by deep packet inspection are available in the web console in the Alerts page. ### Verify deep packet inspection is running[​](#verify-deep-packet-inspection-is-running) Get the [status of DeepPacketInspection](https://docs.tigera.io/calico-cloud/reference/resources/deeppacketinspection#status) resource to verify if live traffic is being monitored on selected workload endpoints. ```bash kubectl get -n ``` ### Install custom Snort rules[​](#install-custom-snort-rules) If you don't want to use the [Snort community rules](https://www.snort.org/downloads/#rule-downloads), you can install a custom set of Snort rules to perform deep packet inspections. You may want to install your own rules if: - you use a paid subscription to a Snort ruleset - you have written your own Snort rules. > **INFO:** If you install custom Snort rules, Calico Cloud will stop updating the community rules with each minor release. You will be responsible for making sure your rules are up to date. 1. Create a container with your custom Snort rules. 1. Copy your Snort rule files into a new `./snort-rules/` directory. 2. Create a `Dockerfile` like this one: ```yaml FROM alpine:3.14 COPY snort-rules /snort-rules ENTRYPOINT [ "/bin/sh", "-c", "cp /snort-rules/* /usr/etc/snort/rules/" ] ``` 3. In the console, run the following commands: ```bash docker build . -t : docker push : ``` After the image has been pushed to the registry, you're ready to configure the `IntrusionDetection` resource. 2. Update the `IntrusionDetection` resource with the custom Snort rules image. ```yaml spec: deepPacketInspectionDaemonset: spec: template: spec: initContainers: - name: snort-rules image: : ``` This can also be done by running the following command: ```bash kubectl patch intrusiondetection tigera-secure --type merge -p '{"spec":{"deepPacketInspectionDaemonset":{"spec":{"template":{"spec":{"initContainers":[{"name":"snort-rules", "image":":"}]}}}}}}' ``` 3. Verify that your custom rules have been installed correctly: 1. If it's not running already, [apply the DeepPacketInspection resource](#configure-deep-packet-inspection) to your cluster. 2. Extract the list of Snort rules that are currently being used by running the following commands: ```bash export POD=$(kubectl get pods -n tigera-dpi -o custom-columns=:metadata.name --no-headers | head -n 1) \ kubectl exec -n tigera-dpi $POD -- tar -cf - /usr/etc/snort/rules | tar --strip-components=4 -xf - ``` 3. If these rules match those in your custom set, then the installation was successful. ## Additional resources[​](#additional-resources) - [Configure packet capture](https://docs.tigera.io/calico-cloud/observability/packetcapture) ### Container threat detection > **WARNING:** Compliance reports are deprecated and will be removed in a future release. We're building a new compliance reporting system that will eventually replace the current one. ## Big picture[​](#big-picture) Get alerts when security threats, such as malware and other suspicious processes, are detected in your cluster. ## Value[​](#value) Calico Cloud provides a threat detection engine that analyzes observed file and process activity to detect known malicious and suspicious activity. As part of these threat detection capabilities, Calico Cloud maintains a database of malware file hashes. This database consists of SHA256, SHA1, and MD5 hashes of executable file contents that are known to be malicious. Whenever a program is launched in a Calico Cloud cluster, malware detection generates an alert in the **Security Events Dashboard** if the program's hash matches one that is known to be malicious. Our threat detection engine also monitors activity within the containers running in your clusters to detect suspicious behavior and generate corresponding alerts. The threat detection engine monitors the following types of suspicious activity within containers: - Access to sensitive system files and directories - Command and control - Defense evasion - Discovery - Execution - Impact - Persistence - Privilege escalation ## Before you begin...[​](#before-you-begin) ### Required[​](#required) Calico Cloud Container threat detection uses eBPF to monitor container activity, and it runs on Linux-based nodes in a Kubernetes cluster. Nodes require amd64 (x86\_64) architecture CPUs and one of the following distributions: - Ubuntu Bionic with kernel version 4.15.0 or 5.4.0 - Ubuntu Focal with kernel version 5.4.0, 5.8.0 or 5.11.0 - CentOS 7 or 8 - Fedora 29, 30, 31, 32, 33 or 34 - Amazon Linux 2 - Debian Stretch or later - Any other distribution with a Linux kernel 5.0 or later that provides BPF Type Format (BTF) for that kernel at the standard place (/sys/kernel/btf/vmlinux) > **SECONDARY:** If your nodes are running a variant kernel, or a similarly-modern kernel but with another platform, please open a [Support ticket](https://support.tigera.io/) so we can bundle the BTF data to precisely match the version of the kernel running on your cluster nodes. ## How to[​](#how-to) - [Enable Container threat detection in the managed cluster](#enable-container-threat-detection) - [Monitor the Security Events page for malicious programs](#monitor-alerts-page-for-malicious-programs) - [Exclude a process from Security Events alerts](#exclude-a-process-from-Security-Events-alerts) - [Update detectors settings](#update-detectors-settings) - [Configure detectors via RuntimeSecurity Custom Resource](#configure-detectors-via-runtimesecurity-custom-resource) ### Enable Container Threat Detection[​](#enable-container-threat-detection) Container threat detection is disabled by default. To enable Container threat detection on your managed cluster, go to the **Threat Defense** section in the Calico Cloud UI, and select **Enable Container Threat Detection**. This will result in Container threat detection running on all nodes in the managed cluster to detect malware and suspicious processes. Alternatively, Container threat detection can be enabled using kubectl: ```bash kubectl apply -f - < **SECONDARY:** This feature is tech preview. Tech preview features may be subject to significant changes before they become GA. ## Big picture[​](#big-picture) Protect cloud-native applications from application layer attacks with Calico Cloud Workload-based Web Application Firewall (WAF). ## Value[​](#value) Our workload-centric Web Application Firewall (WAF) protects your workloads from a variety of application layer attacks originating from within your cluster such as [SQL injection](https://owasp.org/www-community/attacks/SQL_Injection). Given that attacks on apps are the [leading cause of breaches](https://www.f5.com/labs/articles/threat-intelligence/application-protection-report-2019--episode-2--2018-breach-trend), you need to secure the HTTP traffic inside your cluster. Historically, web application firewalls (WAFs) were deployed at the edge of your cluster to filter incoming traffic. Our workload-based WAF solution takes a unique, cloud-native approach to web security by allowing you to implement zero-trust rules for workloads inside your cluster. ## Concepts[​](#concepts) ### About Calico Cloud WAF[​](#about-calico-cloud-waf) WAF is deployed in your cluster with Envoy as a sidecar container on your pods. Calico Cloud proxies traffic through Envoy, checking HTTP requests using the industry-standard [ModSecurity](https://owasp.org/www-project-modsecurity-core-rule-set/) with OWASP Core Rule Set `v4.7.0` with some modifications for Kubernetes workloads. You simply enable WAF in the web console, and determine the deployments that you want to enable for WAF protection. By default WAF is set to `DetectionOnly` so no traffic will be denied until you are ready to turn on blocking mode. Every request that WAF finds an issue with, will result in a Security Event being created for [you to review in the UI](#view-waf-events), regardless of whether the traffic was allowed or denied. This can greatly help in tuning later. #### How WAF determines if a request should be allowed or denied[​](#how-waf-determines-if-a-request-should-be-allowed-or-denied) If you configure WAF in blocking mode, WAF will use something called [anomaly scoring mode](https://coreruleset.org/docs/2-how-crs-works/2-1-anomaly_scoring/) to determine if a request is allowed with `200 OK` or denied `403 Forbidden`. This works by matching a single HTTP request against all the configured WAF rules. Each rule has a score and WAF adds all the matched rule scores together, and compares it to the overall anomaly threshold score (100 by default). If the score is under the threshold the request is allowed and if the score is over the threshold the request is denied. Our WAF starts in detection mode only and with a high default scoring threshold so is safe to turn on and then [fine-tune the WAF](#manage-your-waf) for your specific needs in your cluster. ## Before you begin[​](#before-you-begin) ### Limitations[​](#limitations) WAF cannot be used with: - Host-networked client pods - TLS traffic - [LoadBalancer services](https://Kubernetes.io/docs/concepts/services-networking/service/#loadbalancer) - Kubernetes 1.28 or earlier > **SECONDARY:** When selecting and deselecting traffic for WAF, active connections may be disrupted. > **WARNING:** You should enable WAF only for your application pods. If you enable WAF for deployments that are part of the Kubernetes system itself, you may create an undesirable cluster state. ## Enable WAF[​](#enable-waf) ### Enable WAF using the CLI[​](#enable-waf-using-the-cli) #### Using kubectl[​](#using-kubectl) Create an (or edit an existing) ApplicationLayer resource named `tigera-secure`, set the `sidecarInjection` field to `Enabled`. ```bash kubectl apply -f - < Web Application Firewall > Configure Web Application Firewall**. ![WAF services](https://docs.tigera.io/img/calico-enterprise/waf-config-first-step.png) Then, wait for the Tigera APIServer deployment and Log collection daemonset to progress to ready. ```bash kubectl rollout status -n calico-system deployment/calico-apiserver --timeout=120s kubectl rollout status -n calico-system ds/l7-log-collector --timeout=120s ``` ### (Optional) Deploy a sample application[​](#optional-deploy-a-sample-application) If you don’t have an application to test WAF with or don’t want to use it right away against your own application, we recommend that you install the [GoogleCloudPlatform/microservices-demo app](https://github.com/GoogleCloudPlatform/microservices-demo): ```bash kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/microservices-demo/main/release/kubernetes-manifests.yaml ``` ## Apply WAF to deployments[​](#apply-waf-to-deployments) Now that you have deployed WAF in your cluster, you can select the deployments you want to protect from application layer attacks. If you have deployed the sample application, you can apply WAF on a deployment associated with your app, as follows: ```bash kubectl patch deployment frontend -n default -p '{"spec":{"template":{"metadata":{"labels":{"applicationlayer.projectcalico.org/sidecar":"true"},"annotations":{"applicationlayer.projectcalico.org/waf":"Enabled"}}}}}' ``` Alternatively, you can use the web console to apply WAF to the `frontend` deployment if you opted for sidecars. In this example, we applied WAF to the `frontend` pods. This means that every request that goes through the `frontend` deployment is inspected. However, the traffic is not blocked because the WAF rule is set to `DetectionOnly` by default. You can adjust rules and start blocking traffic by [fine-tuning your WAF](#manage-waf-configuration). In the previous example, we applied WAF to the `frontend` deployment of the sample application. Here, we are applying WAF to a deployment of your own application. 1. On the web console, click **Threat Defense**, **Web Application Firewall**. 2. Select the deployments you want WAF to inspect, and then click **Confirm Selections**. ![WAF services](https://docs.tigera.io/img/calico-enterprise/waf-select-services-3.17.png) 3. On the **Web Application Firewall** page, you can verify that WAF is enabled for a deployment by locating the deployment and checking that the **Status** column says **Enabled**. 4. To make further changes to a deployment, click **Actions**, and then **Enable** or **Disable**. You have now applied WAF rule sets to your own deployments, and note that the traffic that goes through the selected deployments will be alerted but not blocked by default. ### Trigger a WAF event[​](#trigger-a-waf-event) If you would like to trigger a WAF event for testing purposes, you can simulate an SQL injection attack inside your cluster by crafting a HTTP request with a query string that WAF will detect as an SQL injection attempt. The query string in this example has some SQL syntax embedded in the text. This is harmless and for demo purposes, but WAF will detect this pattern and create an event for this HTTP request. Run a simple curl command from any pod inside your cluster targeting a deployment you have selected for WAF protection e.g. from the demo app above we could attempt to send a simple HTTP request to the cartservice. ```text curl http://frontend/cart?artist=0+div+1+union%23foo*%2F*bar%0D%0Aselect%23foo%0D%0A1%2C2%2Ccurrent_user ``` ## View default rule set details from the web console[​](#view-default-rule-set-details-from-the-web-console) You can view the default rules from the Core Rule Set (OWASP Top 10) by using the web console. Viewing the rules gives you an overview of the rule set, and it can be used when customizing your WAF configuration. This method always shows you the default configuration, even if you have added customizations. 1. From the web console, go to **Threat > Web Application Firewall**, and then click the **Rulesets** tab. 2. Click the rule set card. 3. To view details about specific rules, click any item in the list. ## Manage WAF configuration[​](#manage-waf-configuration) Reviewing the default rule set config: ```bash Include coraza.conf Include crs-setup.conf Include coreruleset/*.conf SecRuleEngine DetectionOnly ``` The configuration file starts with importing the appropriate rule set config. We use Coraza WAF's recommended [Core Rule Set setup](https://coraza.io/docs/tutorials/coreruleset/) files: 1. Coraza recommended configuration with [coraza.conf](https://github.com/corazawaf/coraza-coreruleset/blob/main/rules/%40coraza.conf-recommended) and [crs-setup.conf](https://github.com/corazawaf/coraza-coreruleset/blob/main/rules/%40crs-setup.conf.example) (kept in the same ConfigMap) 2. The rest of the [coreruleset](https://github.com/corazawaf/coraza-coreruleset/tree/main/rules/%40owasp_crs) files, currently [v4.7.0](https://github.com/corazawaf/coraza-coreruleset/tree/v4.7.0/rules/%40owasp_crs) These files can be customized if desired. Add all your customizations directly under `tigera.conf`: ```bash kubectl edit cm -n tigera-operator tigera-waf-config ``` After editing this ConfigMap successfully, the `tigera-waf-config` ConfigMap will be replaced in the `tigera-operator` namespace, which then triggers a rolling restart of your L7 pods. This means that the HTTP connections going through L7 pods at the time of pod termination will be (RST) reset. > **SECONDARY:** It is important to adhere to the [Core Rule Set documentation](https://coreruleset.org/docs) on how to edit the behaviour of your WAF. A good place to begin at is the [Installing Core Rule Set](https://coreruleset.org/docs/1-getting-started/1-1-crs-installation/). > > In many scenarios, the default example CRS configuration will be a good enough starting point. It is recommended to review the example configuration file before you deploy it to make sure it’s right for your environment. ### Customization options[​](#customization-options) #### Set WAF to block traffic[​](#set-waf-to-block-traffic) By default WAF will not block a request even if it has matching rule violations. The rule engine is set to `DetectionOnly`. You can configure to block traffic instead with an `HTTP 403 Forbidden` response status code when the combined matched rules scores exceed a certain threshold. 1. Edit the configmap: ```bash kubectl edit cm -n tigera-operator tigera-waf-config ``` 2. Look for `SecRuleEngine DetectionOnly` and change it to `SecRuleEngine On`. 3. Save your changes. This triggers a rolling update of the L7 pods. | Action | Description | Disruptive? | | ------------- | ------------------------------------------------------------------------------------ | ----------- | | DetectionOnly | Traffic is not denied nor dropped. Calico Cloud will log events. | No | | On | Denies HTTP traffic. Calico Cloud will log the event in Security Events. | Yes | | Off | Be cautious about using this option. Traffic is not denied, and there are no events. | No | #### Other basic customizations[​](#other-basic-customizations) For basic customizations, it's best to add it after all the includes in `tigera.conf`. In fact, this is the reason why the `SecRuleEngine` directive and the rest of [our customizations](https://github.com/tigera/operator/blob/master/pkg/render/applicationlayer/ruleset/coreruleset/tigera.conf#L8-L17) are situated there. An example is adding a sampling mode. For that, the `tigera.conf` will look like this: ```bash # CoreRuleset activation Include coraza.conf Include crs-setup.conf Include coreruleset/*.conf SecRuleEngine DetectionOnly # --- all customizations appear below this line, unless they need a specific loading order like plugins --- # --- Add sampling mode # Read about sampling mode here https://coreruleset.org/docs/2-how-crs-works/2-4-sampling_mode/ SecAction "id:900400,\ phase:1,\ pass,\ nolog,\ setvar:tx.sampling_percentage=50" ``` Also you can disable certain rules here: ```bash # --- disable 'Request content type is not allowed by policy' SecRuleRemoveById 920420 ``` Change anomaly scoring threshold: ```bash SecAction \ "id:900110,\ phase:1,\ nolog,\ pass,\ t:none,\ setvar:tx.inbound_anomaly_score_threshold=25,\ setvar:tx.outbound_anomaly_score_threshold=20" ``` Or even change rule action parameters or behavior. For example: ```bash # --- append to more allowed content types to request bodies SecAction \ "id:900220,\ phase:1,\ nolog,\ pass,\ t:none,\ setvar:'tx.allowed_request_content_type=|application/x-www-form-urlencoded| |multipart/form-data| |multipart/related| |text/xml| |application/xml| |application/soap+xml| |application/json| |application/cloudevents+json| |application/cloudevents-batch+json| |application/grpc|'" ``` #### Using Core Rule Set plugins[​](#using-core-rule-set-plugins) Let's go with an example plugin: [Wordpress Rule Exclusions](https://github.com/coreruleset/wordpress-rule-exclusions-plugin/). Plugin files are the following: ```text wordpress-rule-exclusions-before.conf wordpress-rule-exclusions-config.conf ``` To include these files properly, structure your work directory like so: ```text tigera.conf coraza.conf crs-setup.conf wordpress-rule-exclusions-before.conf wordpress-rule-exclusions-config.conf ``` and then `tigera.conf` contents should be: ```bash Include coraza.conf Include wordpress-rule-exclusions-config.conf Include wordpress-rule-exclusions-before.conf Include crs-setup.conf Include coreruleset/*.conf # if your plugin has an -after.conf, include them here # but wordpress rule exclusions doesn't so we're adding a comment placeholder # Include wordpress-rule-exclusions-after.conf SecRuleEngine DetectionOnly ``` Then create and apply the configmap: ```bash ## create the configuration map itself kubectl create cm --dry-run=client \ --from-file=tigera.conf \ --from-file=coraza.conf \ --from-file=crs-setup.conf \ --from-file=wordpress-rule-exclusions-config.conf \ --from-file=wordpress-rule-exclusions-before.conf \ -n tigera-operator tigera-waf-config -o yaml > ruleset.configmap.yaml ## replace active configmap kubectl replace -f ruleset.configmap.yaml ``` Read more about the order of execution for plugins here: [https://coreruleset.org/docs/4-about-plugins/4-1-plugins/](https://coreruleset.org/docs/4-about-plugins/4-1-plugins/) ## View WAF events[​](#view-waf-events) ### Security Events[​](#security-events) To view WAF events in a centralized security events dashboard, go to: **Threat defense**, **Security Events**. For help, see [Security Event Management](https://docs.tigera.io/calico-cloud/threat/security-event-management). ### Kibana[​](#kibana) To view WAF events In Kibana, select the `tigera_secure_ee_waf*` index pattern. ### Disable WAF for a deployment[​](#disable-waf-for-a-deployment) To disable WAF on a deployment, use the Actions menu on the WAF board, or use the following command: ```bash kubectl patch deployment -n --type='json' -p '[{"op":"remove","path":"/spec/template/metadata/labels/applicationlayer.projectcalico.org~1sidecar"},{"op":"remove","path":"/spec/template/metadata/annotations/applicationlayer.projectcalico.org~1waf"}]' ``` ## Disable WAF feature from your cluster[​](#disable-waf-feature-from-your-cluster) To safely disable WAF, determine how to handle ApplicationLayer features and follow the steps: ### Disable all ApplicationLayer features, including WAF[​](#disable-all-applicationlayer-features-including-waf) ```bash kubectl delete applicationlayer tigera-secure ``` ### Keep some ApplicationLayer features enabled[​](#keep-some-applicationlayer-features-enabled) To disable WAF but keep some ApplicationLayer features enabled, you must update the [ApplicationLayer](https://docs.tigera.io/calico-cloud/reference/installation/api#applicationlayer) custom resource. Note that the [ApplicationLayer Specification](https://docs.tigera.io/calico-cloud/reference/installation/api#applicationlayerspec) can specify configuration for [application logging](https://docs.tigera.io/calico-cloud/reference/installation/api#operator.tigera.io/v1.logcollectionspec) and [application layer policy](https://docs.tigera.io/calico-cloud/reference/installation/api#operator.tigera.io/v1.applicationlayerpolicystatustype) also. For the ApplicationLayer custom resource to be valid, at least one of these features have to be enabled, for example: #### Valid YAML[​](#valid-yaml) WAF disabled, ALP enabled (using per-host proxy), and Log collection is unspecified (and the default is disabled) ```yaml apiVersion: operator.tigera.io/v1 kind: ApplicationLayer metadata: name: tigera-secure spec: webApplicationFirewall: Disabled applicationLayerPolicy: Enabled sidecarInjection: Disabled ``` ##### Invalid YAML[​](#invalid-yaml) WAF and ALP both set as disabled, log collection is unspecified (and the default is disabled) ```yaml apiVersion: operator.tigera.io/v1 kind: ApplicationLayer metadata: name: tigera-secure spec: webApplicationFirewall: Disabled applicationLayerPolicy: Disabled sidecarInjection: Disabled ``` ### Deploy a web application firewall with Calico Ingress Gateway This guide shows you how to deploy and configure a web application firewall (WAF) with the Calico Ingress Gateway. By deploying the WAF in this way, you can protect publicly exposed services from a variety of application-layer attacks. > **SECONDARY:** This feature is tech preview. Tech preview features may be subject to significant changes before they become GA. ## Prerequisites[​](#prerequisites) - [Calico Ingress Gateway](https://docs.tigera.io/calico-cloud/networking/ingress-gateway/about-calico-ingress-gateway) is set up for your cluster. - You have the value of `metadata.name` for the `Gateway` resource you want to use with WAF. ## Deploy WAF on Calico Ingress Gateway[​](#deploy-waf-on-calico-ingress-gateway) Calico Ingress Gateway uses Envoy's implementation of the Kubernetes Gateway API. To direct ingress gateway traffic through the WAF, you need to configure Envoy to use the WAF HTTP filter. To enable WAF on a Calico Ingress Gateway: 1. Create a `Backend` resource to allow Envoy to find the WAF HTTP filter: ```bash kubectl apply -f - < EOF ``` Replace `` with the value of `metadata.name` in your `Gateway` resource. The WAF now processes all HTTP requests that pass through the specified `Gateway` resource. To deploy a WAF on multiple gateways, you must create a separate `EnvoyExtensionPolicy` resource for each `Gateway` resource. Each `EnvoyExtensionPolicy` must reference the same `tigera-waf-backend` backend. 3. To verify that the WAF is enabled for your gateway, you can simulate an SQL injection attack through your gateway and see whether it triggers a security event. > **SECONDARY:** The query string in this example has some SQL syntax embedded in the text. This is harmless and for demo purposes, but WAF will detect this pattern and create an WAF log for this HTTP request. 1. Get the service IP of your gateway: ```bash export GATEWAY_HOST=$(kubectl get gateway/ -o jsonpath='{.status.addresses[0].value}') ``` 2. Trigger a security event by simulating an SQL injection attack on that service IP: ```bash curl --verbose --header "Host: www.example.com" http://$GATEWAY_HOST/?artist=0+div+1+union%23foo*%2F*bar%0D%0Aselect%23foo%0D%0A1%2C2%2Ccurrent_user ``` 3. From the web console, go to **Threat > Security Events** and check for a security event with corresponds with this simulated SQL injection attack. ## Customizing your WAF configuration for an ingress gateway[​](#customizing-your-waf-configuration-for-an-ingress-gateway) The default WAF configuration uses the OWASP Core Rule Set v4.7.0 in detection-only mode with early blocking enabled. To customize a WAF configuration for an ingress gateway, you can include custom directives in an `EnvoyPatchPolicy` resource. For information on how to create custom directives, see [WAF customization options](https://docs.tigera.io/calico-cloud/threat/web-application-firewall#customization-options). For example, to enable the OWASP Core Rule Set in blocking mode, you can apply an `EnvoyPatchPolicy` resource like this: ```yaml apiVersion: gateway.envoyproxy.io/v1alpha1 kind: EnvoyPatchPolicy metadata: name: tigera-waf-http-filter-metadata namespace: spec: targetRef: group: gateway.networking.k8s.io kind: Gateway name: type: JSONPatch jsonPatches: - type: "type.googleapis.com/envoy.config.listener.v3.Listener" name: //http operation: op: add jsonPath: "default_filter_chain.filters[0].typed_config.http_filters[?match(@.name, '.*tigera-waf-envoy-extension-policy/.*')]" path: typed_config/grpc_service/initial_metadata value: - key: directivesJson value: "[\"SecRuleEngine On\"]" ``` You can add additional directives to the `directivesJson` array to further customize the WAF behavior. Make sure that combine the directives as an inlined JSON list with proper escaping. The array in the following snippet is an example of how to set the WAF to detection-only mode, remove a specific rule, and set a sampling percentage: ```yaml - key: directivesJson value: "[\"SecRuleEngine DetectionOnly\", \"SecRuleRemoveById 920420\", \"SecAction \\\""id:900400,phase:1,pass,nolog,setvar:tx.sampling_percentage=50\\\"\]" ``` ## Additional resources[​](#additional-resources) - [OWASP ModSecurity](https://modsecurity.org/) - [OWASP Core Rule Set documentation](https://coreruleset.org/) ### Image Assurance > **WARNING:** This feature was deprecated in Calico Cloud version 21.1.0 and will be removed in a future release. Availability depends on when you started using Calico Cloud. > > - For users who started using Calico Cloud in April 2025 or later, this feature is not available. > - Legacy users, who started using Calico Cloud before April 2025, can continue to use this feature until it is removed in a future release. Detect and block vulnerable images from container workloads. ## Scaning images for vulnerabilities[​](#scaning-images-for-vulnerabilities) ##### [Choose an image scanning method](https://docs.tigera.io/calico-cloud/image-assurance/scanners/overview) [Choose a method to scan images for vulnerabilities.](https://docs.tigera.io/calico-cloud/image-assurance/scanners/overview) ##### [Scan images in a Kubernetes cluster](https://docs.tigera.io/calico-cloud/image-assurance/scanners/cluster-scanner) [Detect vulnerabilities in a Kubernetes cluster.](https://docs.tigera.io/calico-cloud/image-assurance/scanners/cluster-scanner) ##### [Scan images in container registries](https://docs.tigera.io/calico-cloud/image-assurance/scanners/registry-scanner) [Scan images in container registries.](https://docs.tigera.io/calico-cloud/image-assurance/scanners/registry-scanner) ##### [Integrate the scanner into your build pipeline](https://docs.tigera.io/calico-cloud/image-assurance/scanners/pipeline-scanner) [Scan images in your build pipeline using Image Assurance.](https://docs.tigera.io/calico-cloud/image-assurance/scanners/pipeline-scanner) ## Working with scan results[​](#working-with-scan-results) ##### [Set up alerts on vulnerabilities](https://docs.tigera.io/calico-cloud/image-assurance/set-up-alerts) [Get alerts on vulnerabilities.](https://docs.tigera.io/calico-cloud/image-assurance/set-up-alerts) ##### [View scanned and running images](https://docs.tigera.io/calico-cloud/image-assurance/understanding-scan-results) [Understand scan results in the web console.](https://docs.tigera.io/calico-cloud/image-assurance/understanding-scan-results) ##### [Exclude vulnerabilities from scan results](https://docs.tigera.io/calico-cloud/image-assurance/exclude-vulnerabilities-from-scan-results) [Reduce noise in your Image Assurance scan results by excluding vulnerabilities.](https://docs.tigera.io/calico-cloud/image-assurance/exclude-vulnerabilities-from-scan-results) ### Scanners > **WARNING:** This feature was deprecated in Calico Cloud version 21.1.0 and will be removed in a future release. Availability depends on when you started using Calico Cloud. > > - For users who started using Calico Cloud in April 2025 or later, this feature is not available. > - Legacy users, who started using Calico Cloud before April 2025, can continue to use this feature until it is removed in a future release. ## [📄️ Choose an image scanning method](https://docs.tigera.io/calico-cloud/image-assurance/scanners/overview) [Choose a method to scan images for vulnerabilities.](https://docs.tigera.io/calico-cloud/image-assurance/scanners/overview) ## [📄️ Scan images in a Kubernetes cluster](https://docs.tigera.io/calico-cloud/image-assurance/scanners/cluster-scanner) [Detect vulnerabilities in a Kubernetes cluster.](https://docs.tigera.io/calico-cloud/image-assurance/scanners/cluster-scanner) ## [📄️ Integrate the scanner into your build pipeline](https://docs.tigera.io/calico-cloud/image-assurance/scanners/pipeline-scanner) [Scan images in your build pipeline using Image Assurance.](https://docs.tigera.io/calico-cloud/image-assurance/scanners/pipeline-scanner) ## [📄️ Scan images in container registries](https://docs.tigera.io/calico-cloud/image-assurance/scanners/registry-scanner) [Scan images in container registries.](https://docs.tigera.io/calico-cloud/image-assurance/scanners/registry-scanner) ### Choose an image scanning method > **WARNING:** This feature was deprecated in Calico Cloud version 21.1.0 and will be removed in a future release. Availability depends on when you started using Calico Cloud. > > - For users who started using Calico Cloud in April 2025 or later, this feature is not available. > - Legacy users, who started using Calico Cloud before April 2025, can continue to use this feature until it is removed in a future release. ## Big picture[​](#big-picture) Scan images and Kubernetes workloads for vulnerabilities using Calico Cloud Image Assurance. ## Value[​](#value) Calico Cloud Image Assurances helps you identify vulnerabilities in container images that you deploy to Kubernetes clusters. Vulnerabilities are known flaws in libraries and packages used by applications that attackers can exploit and cause harm. With Image Assurance you can: - Scan an image for vulnerabilities - Assess the impact of newly-found vulnerabilities and prioritize remediation efforts - Catch vulnerabilities days or weeks later with continuous image rescanning - Create exceptions to ignore specific vulnerabilities - Create alerts on high-severity vulnerabilities so you can delegate remediation efforts to the appropriate team - Block non-compliant workloads using policy as part of your cloud-native security posture ## About Image Assurance[​](#about-image-assurance) Image Assurance is based on the Common Vulnerabilities and Exposures (CVE) system, which provides a catalog of publicly-known security vulnerabilities and exposures. Known vulnerabilities are identified by a unique CVE ID based on the year it was reported (for example, CVE-2021-44228). Scanned image content includes: - Libraries and content (for example, python, ruby gems, jars and go) - Packages (OS and non-OS) - Image layer ## Image scanning options[​](#image-scanning-options) Image Assurance provides different versions of the scanner to accommodate different use cases as shown in the following table. | Scan images in... | Description | Scanner access | Benefits | | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Kubernetes cluster](https://docs.tigera.io/calico-cloud/image-assurance/scanners/cluster-scanner) | Scan any running image in the Kubernetes cluster including locally-built first-party images to fix critical bugs. | Runs automatically in the managed cluster in the web console | The Image Assurance dashboard provides an easy way to get started with vulnerability scanning and remediation, and defense-in-depth coverage without building your own scanning solution. | | [CI/CD pipeline](https://docs.tigera.io/calico-cloud/image-assurance/scanners/pipeline-scanner) | Integrate the CLI scanner in your application build pipeline and private registries including: - Customer-built images - Local images - Third-party images from public registries (for example Kafka, Redis) | A downloadable binary | Incorporate the scanner as a lightweight runner in your build pipeline. Use the scanner offline and on-demand for ad hoc scanning and emergency patching. | | [Image registries](https://docs.tigera.io/calico-cloud/image-assurance/scanners/registry-scanner) | Scan images in registries (for example, Amazon ECR). | A downloadable Docker image | Add a layer of defense for images that were not scanned in your build pipeline, but get published to your registry. | ### Scan images in a Kubernetes cluster > **WARNING:** This feature was deprecated in Calico Cloud version 21.1.0 and will be removed in a future release. Availability depends on when you started using Calico Cloud. > > - For users who started using Calico Cloud in April 2025 or later, this feature is not available. > - Legacy users, who started using Calico Cloud before April 2025, can continue to use this feature until it is removed in a future release. Scan all images in a Kubernetes cluster for vulnerabilities to achieve a continuous clean bill of health and defense in depth. Common use cases for scanning in a Kubernetes cluster are: - Images may pass scanning during the build phase, but they could contain vulnerabilities days or weeks later - Third-party images that are pulled from public registries are often not scanned in build pipelines and can contain Critical or High vulnerabilities - Application teams may build one-off images outside of their pipeline to make an emergency patch and fix a critical bug. ## About Image Assurance scanner[​](#about-image-assurance-scanner) The Image Assurance scanner that runs in a Kubernetes cluster is out-of-the-box ready to use without configuration. It runs as a daemonset in a managed cluster where images are located, and is installed on all nodes in the cluster. Vulnerability detection consists of these steps: - **Image Assurance scanner** - generates a dependency list Software Bill of Materials (SBOM) using Syft - **Vulnerability lookup** - Calico Cloud uploads the SBOM where packages are matched with known CVEs in the vulnerability databases based on dependencies using Grype ![vulnerability-detection](https://docs.tigera.io/assets/images/vulnerability-detection-efcee2ac7c221b5b55198cf5dee97b7f.png) Calico Cloud checks running images for new vulnerabilities every 24 hours and reports scan results to the Image dashboard in the web console. ## Before you begin[​](#before-you-begin) **Unsupported** - OpenShift - GCP-Kubeadm - AWS-Kubeadm - GKE **Cluster requirements** - Containerd is the container runtime - AKS clusters: if you are using Kubernetes v1.19 or higher, containerd should be your default runtime - Containerd must be using overlays or native file system snapshotter ## How to[​](#how-to) - [Get latest version of Image Assurance](#get-latest-version-of-image-assurance) - [Enable scanner](#enable-scanner) - [Customize scanner settings](#customize-scanner-settings) - [Disable scanner](#disable-scanner) ### Get latest version of Image Assurance[​](#get-latest-version-of-image-assurance) 1. On the **Managed Clusters** page, select the cluster from the list, and click **Reinstall**. 2. Copy the updated installation script command and run it against your cluster. ### Enable scanner[​](#enable-scanner) Complete the following steps for each managed cluster you want enabled with the cluster scanner: 1. Modify the [Image Assurance](https://docs.tigera.io/calico-cloud/reference/installation/ia-api#image-assurance.operator.tigera.io/v1.ImageAssuranceSpec) installation resource. ```bash kubectl edit imageassurance default ``` 2. Set the `clusterScanner` field to `Enabled` and save the file. The cluster scanner is deployed as a container inside the `tigera-image-assurance-crawdad` daemonset. 3. Verify that a new container with name, `cluster-scanner` is created inside the daemonset. That’s it. The cluster scanner will start scanning images on running pods in the cluster. For help viewing image events in the web console, see [View scanned and running images](https://docs.tigera.io/calico-cloud/image-assurance/understanding-scan-results). ### Customize scanner settings[​](#customize-scanner-settings) To change default settings, modify the [Image Assurance](https://docs.tigera.io/calico-cloud/reference/installation/ia-api#image-assurance.operator.tigera.io/v1.ImageAssuranceSpec) installation resource. - Container runtime socket path Set the `criSocketPath` field to the path of the container runtime socket. Default: `/run/containerd/containerd.sock` - Containerd file system root path Set the `containerdVolumeMountPath`. Default: `/var/lib/containerd/`. ### Configure exclusions for image scanning[​](#configure-exclusions-for-image-scanning) To specify which namespaces should be excluded from future scans, follow the following steps. - Modify your [Image Assurance](https://docs.tigera.io/calico-cloud/reference/installation/ia-api#image-assurance.operator.tigera.io/v1.ImageAssuranceSpec) installation resource to include the `exclusions.namespaces` field. List each namespace you want to exclude. ```yaml apiVersion: image-assurance.operator.tigera.io/v1 kind: ImageAssurance metadata: name: default spec: clusterScanner: Enabled exclusions: namespaces: - "kube-system" - "dev-qa" ``` In this example, the workloads in the `kube-system` and `dev-qa` namespaces are excluded from future image scans. > **SECONDARY:** Applying or updating namespace exclusions affects only future scans. Results from scans conducted prior to these exclusions will remain unchanged. The exclusions configured will apply to both cluster scanner and runtime view (**Running Images** tab). ### Disable scanner[​](#disable-scanner) 1. Modify the `imageassurance` installation resource. ```bash kubectl edit imageassurance default ``` 2. Set the `clusterScanner` field to `Disabled` and save the file. This deletes the cluster scanner container from the daemonset from your cluster. ## Next step[​](#next-step) [Set up alerts on vulnerabilities](https://docs.tigera.io/calico-cloud/image-assurance/set-up-alerts) ### Integrate the scanner into your build pipeline > **WARNING:** This feature was deprecated in Calico Cloud version 21.1.0 and will be removed in a future release. Availability depends on when you started using Calico Cloud. > > - For users who started using Calico Cloud in April 2025 or later, this feature is not available. > - Legacy users, who started using Calico Cloud before April 2025, can continue to use this feature until it is removed in a future release. ## Big picture[​](#big-picture) Integrate the Image Assurance CLI scanner into your build pipeline to ensure builds are checked by Image Assurance before deployment. ## Value[​](#value) The Image Assurance CLI scanner allows you to manually scan container images locally or remotely for on-demand scanning and emergency scanning. The CLI scanner is ideal for use in a CI/CD pipeline to automatically scan images before pushing them to a registry. If the CLI scanner is part of your pipeline, scanning is done before runtime and the results are displayed in the Image Assurance dashboard in the web console. ## Before you begin[​](#before-you-begin) **Image requirements** - Docker container runtime - Images must be available locally through the Docker container runtime environment where the Image Assurance scanner is running. **Scanner requirements** - Must have internet access to download and update the vulnerability database - To see image scan results in the web console, the scanner must communicate with an external API endpoint outside your environment ## How to[​](#how-to) - [Get the latest version of Image Assurance](#get-the-latest-version-of-image-assurance) - [Start the scanner](#start-the-scanner) - [Integrate the scanner in your build pipeline](#integrate-the-scanner-in-your-build-pipeline) - [Manually scan images](#manually-scan-images) - [Scan images using a configuration file](#scan-images-using-a-configuration-file) ### Get the latest version of Image Assurance[​](#get-the-latest-version-of-image-assurance) 1. On the **Managed Clusters** page, select the cluster from the list, and click **Reinstall**. 2. Copy the updated installation script command and run it against your cluster. ### Start the scanner[​](#start-the-scanner) 1. Download the latest version of the scanner. **Linux** ```shell curl -Lo tigera-scanner https://installer.calicocloud.io/tigera-scanner/v3.22.0-2.0-7/image-assurance-scanner-cli-linux-amd64 ``` **macOS** ```shell curl -Lo tigera-scanner https://installer.calicocloud.io/tigera-scanner/v3.22.0-2.0-7/image-assurance-scanner-cli-darwin-amd64 ``` 2. Set the executable flag on the binary. ```shell chmod +x ./tigera-scanner ``` > **SECONDARY:** You must download and set the executable flag each time you get a new version of the scanner. 3. Verify that the scanner works correctly by running the version command. ```shell ./tigera-scanner version v1.22.7 ``` ### Integrate the scanner into your build pipeline[​](#integrate-the-scanner-into-your-build-pipeline-1) You can include the CLI scanner in your CI/CD pipelines (for example, Jenkins, GitHub actions). Ensure the following: - Download the CLI scanner binary onto your CI runner - If you are running an ephemeral environment in the pipeline, include the download, and update the executable steps in your pipeline to download the scanner on every execution - Create a secret containing the API-Token and API URL and make it available in the pipeline (for example, using a SECURE\_API\_TOKEN environment variable) - Add a step in your pipeline to run the `image-assurance-scanner` after building the container image, and specify the image name as a parameter. For example: `./image-assurance-cli-scanner --apiurl ${IMAGE_NAME}` If your CI platform supports it, you can also use the containerized version of Image Assurance scanner for integrations with other tools like Harness. To integrate the containerized version of Image Assurance scanner into your CI/CD platform, go to: [Image Assurance containerized scanner](https://quay.io/repository/tigera/image-assurance-scanner-cli) and pull the latest image. For example: ```bash docker pull quay.io/tigera/image-assurance-scanner-cli:vx.x.x ``` ### Manually scan images[​](#manually-scan-images) You can scan images and report results back to Calico Cloud, or scan images locally without reporting results to Calico Cloud. **Syntax**: `tigera-scanner scan [OPTIONS] ` **Options**: - `--apiurl` - Calico Cloud API URL path. You can get this URL in the web console, **Image Assurance**, **Scan settings**. - `--token` - secure API or authorization token to make requests to Calico Cloud API URL. You can get this URL in the web console, **Image Assurance**, **Scan settings**. - `--warn_threshold` - CVSS threshold for Warn scan results. Range from 0.0 - 10.0. - `--fail_threshold` - CVSS threshold for Fail scan results. Range from 0.0 - 10.0. - `--vulnerability_db_path` - path to a folder to store vulnerability data (defaults to `$XDG_CACHE_HOME`; if it is not set, defaults to `$HOME/.cache`). - `--input_file ` - Path to a JSON file containing image URLs. - `--output_file ` - File path that will contain scan results in a JSON format. **Examples** **Scan an image, report results** ```shell ./tigera-scanner scan ubuntu:latest --apiurl https://.calicocloud.io --token ezBhbGcetc... ``` **Scan an image locally, do not report results** ```shell ./tigera-scanner scan ubuntu:latest ``` **Scan an image with a failure and warning threshold** ```shell ./tigera-scanner scan ubuntu:latest --fail_threshold 7.0 --warn_threshold 3.9 ``` **Scan multiple images locally, do not report results** ```shell ./tigera-scanner scan ubuntu:latest alpine:latest ``` **Scan multiple images using an input and output file** The input file must have the following JSON structure: ```json { "images": [ "ubuntu:latest", "alpine:latest" ] } ``` ```shell ./tigera-scanner scan --input_file images.json --output_file results.json ``` ### Scan images using a configuration file[​](#scan-images-using-a-configuration-file) Create a configuration file in `$HOME/.tigera-scanner.yaml` for the scanner to read. > **SECONDARY:** Key names must match the full name of arguments passed to the scanner. The configuration precedence order is options > environment variables > file configuration. **Options** | Options | Shorthand | Environment variable | Description | | ------------------------- | --------- | --------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | --apiurl | -a | CC\_API\_URL | Calico Cloud API URL path. You can get this URL in the web console, Image Assurance, Scan settings. | | --token | -t | CC\_TOKEN | Secure API or authorization token to make requests to Calico Cloud API URL. | | --warn\_threshold | -w | CC\_WARN\_THRESHOLD | CVSS threshold for Warn scan results. Range from 0.0 - 10.0. | | --fail\_threshold | -f | CC\_FAIL\_THRESHOLD | CVSS threshold for Fail scan results. Range from 0.0 - 10.0. | | --vulnerability\_db\_path | -p | CC\_VULNERABILITY\_DB\_PATH | Path to a folder to store vulnerability data (defaults to `$XDG_CACHE_HOME`; if it is not set, defaults to `$HOME/.cache`). | | --input\_file | -i | CC\_INPUT\_FILE | Path to the JSON file containing image URLs. | | --output\_file | -o | CC\_OUTPUT\_FILE | File path that will contain scan results in a JSON format. | ## Next step[​](#next-step) [Set up alerts](https://docs.tigera.io/calico-cloud/image-assurance/set-up-alerts) ### Scan images in container registries > **WARNING:** This feature was deprecated in Calico Cloud version 21.1.0 and will be removed in a future release. Availability depends on when you started using Calico Cloud. > > - For users who started using Calico Cloud in April 2025 or later, this feature is not available. > - Legacy users, who started using Calico Cloud before April 2025, can continue to use this feature until it is removed in a future release. Scan images in container registries at any time, on any infrastructure, including Kubernetes. ## Value[​](#value) Add a layer of defense for images that don’t go through a pipeline (for example, third-party images), but are published to a registry. If CVEs are missed in your build pipeline, you can catch them before they are deployed. ## Concepts[​](#concepts) You can run the registry scanner wherever there is a container runtime. And it doesn’t have to be for Kubernetes. To use registry scanner, all you need to do is: - Specify the registry paths to the images you want to scan - Provide permissions for the scanner to access your registries - Get a token for access to the Image Assurance API Based on the paths you specify, the scanner recursively scans all images in the registry once, and sends results to the Image Assurance dashboard in the web console. To deploy the registry scanner as a pod in Kubernetes cluster, we recommend that you define a Kubernetes job or cronjob. ## Before you begin[​](#before-you-begin) **Required** - Registry scanner is running where there is a container runtime - Valid registry credentials **Supported registry platforms** - Amazon Elastic Container Registry (ECR) - Azure Container Registry (ACR) - Google Container Registry (GCR) **Limitations** - The registry scanner is available as an image (not using Tigera Operator installation) - You can scan images from only one of the supported registry platforms/account per instance - If you use the registry scanner with Docker, only tagged images are scanned. However, if you use the scanner with Amazon or Azure, all images (tagged and untagged) are scanned. ## How to[​](#how-to) - [Download the registry scanner](#download-the-registry-scanner) - [Set up authentication to registry scanner](#set-up-authentication-to-registry-scanner) - [Set up registry scanner](#set-up-registry-scanner) - [Scan images and send results to Calico Cloud](#scan-images-and-send-results-to-calico-cloud) - [Troubleshoot](#troubleshoot) - [Get previous registry scanner versions](#get-previous-registry-scanner-versions) ### Download the registry scanner[​](#download-the-registry-scanner) The registry scanner comes in a Docker image. To get the image, run this command: `docker pull quay.io/tigera/image-assurance-registry-scanner:v1.22.7`. ### Set up authentication to registry scanner[​](#set-up-authentication-to-registry-scanner) The registry scanner requires authentication to access a registry. Set up credentials for one of the following registry platforms: - **Docker/Google** - scans only tagged images; untagged images residing in your image registry are not pulled and scanned. - **Azure or AWS** - scans tagged and untagged images (all) #### Docker/GCR required credentials[​](#dockergcr-required-credentials) - `DOCKER_USERNAME` - `DOCKER_PASSWORD` If you have a valid /.docker/config.json, you can also mount this config file on the container while running the registry scanner. ```bash docker run -e ... -v ~/.docker/config.json:/.docker/config.json quay.io/tigera/image-assurance-registry-scanner:v1.22.7 ``` #### Azure required credentials[​](#azure-required-credentials) Registry instances are scanned one at a time. If Docker credentials are found, they are ignored. - `AZURE_CLIENT_ID` - `AZURE_CLIENT_SECRET` or `AZURE_FEDERATED_TOKEN` - `AZURE_TENANT_ID` #### AWS required credentials[​](#aws-required-credentials) Registry instances are scanned one at a time. If Docker credentials are found, they are ignored. - `AWS_ACCESS_KEY_ID` - `AWS_SECRET_ACCESS_KEY` - `AWS_REGION` ### Set up registry scanner[​](#set-up-registry-scanner) **Required** - `REGISTRY` - the registry you want to scan. For example, gcr.io. - `IMAGE_ASSURANCE_API_URL` - Get the URL in the web console - `IMAGE_ASSURANCE_API_TOKEN` - Get the token in the web console - Registry credentials: Docker/gcr, acr, or ecr **Optional** `REGISTRY_FILTER` - limits scanning time when you have thousands of repositories and images. Supports a comma-separated list. Example: gcr registry ```bash gcr.io/prod-env/api gcr.io/staging-env/api gcr.io/dev/api ``` To filter out images in the dev "sub" registry: `-e REGISTRY_FILTER=prod-env,staging-env` ### Scan images and send results to Calico Cloud[​](#scan-images-and-send-results-to-calico-cloud) Example: gcr registry with Docker credentials ```bash docker run -e REGISTRY=gcr.io -e IMAGE_ASSURANCE_API_URL=https://-management.dev.calicocloud.io/bast -e IMAGE_ASSURANCE_API_TOKEN=$TOKEN -e DOCKER_USERNAME= -e DOCKER_PASSWORD= quay.io/tigera/image-assurance-registry-scanner:v1.22.7 ``` Example: acr registry with Azure credentials ```bash docker run -e REGISTRY=your-org.azurecr.io -e IMAGE_ASSURANCE_API_URL=https://-management.dev.calicocloud.io/bast -e IMAGE_ASSURANCE_API_TOKEN=$TOKEN -e AZURE_CLIENT_ID= -e AZURE_CLIENT_SECRET= -e AZURE_TENANT_ID= quay.io/tigera/image-assurance-registry-scanner:v1.22.7 ``` Example: ecr registry with AWS credentials ```bash docker run -e REGISTRY=gcr.io -e IMAGE_ASSURANCE_API_URL=https://-management.dev.calicocloud.io/bast -e IMAGE_ASSURANCE_API_TOKEN=$TOKEN -e AWS_ACCESS_KEY_ID= -e AWS_SECRET_ACCESS_KEY= -e AWS_REGION= quay.io/tigera/image-assurance-registry-scanner:v1.22.7 ``` Example: run when mounting the dockerfile ```bash docker run -e REGISTRY=gcr.io -e IMAGE_ASSURANCE_API_URL=https://-management.dev.calicocloud.io/bast -e IMAGE_ASSURANCE_API_TOKEN=$TOKEN -v ~/.docker/config.json/:/.docker/config.json quay.io/tigera/image-assurance-registry-scanner:v1.22.7 ``` ### Troubleshoot[​](#troubleshoot) **Issues authenticating with registry scanner** Verify that the registry credentials are correct. **Scan results are not uploading to Calico Cloud** Image scan results that are uploaded to Calico Cloud through the registry scanner require additional processing before appearing in the Image Assurance dashboard. This may result in a time delay before CVE results appear for those images in the UI. You can also verify that the API token and URL are correct. **Scanned images do not match what I expect** Verify that the credentials on the registry side have the correct permission level. ### Get previous registry scanner versions[​](#get-previous-registry-scanner-versions) For previous versions of registry scanner, see [quay repository](https://quay.io/organization/tigera). ## Next step[​](#next-step) - [Understand scan results in the web console](https://docs.tigera.io/calico-cloud/image-assurance/understanding-scan-results) ### View scanned and running images > **WARNING:** This feature was deprecated in Calico Cloud version 21.1.0 and will be removed in a future release. Availability depends on when you started using Calico Cloud. > > - For users who started using Calico Cloud in April 2025 or later, this feature is not available. > - Legacy users, who started using Calico Cloud before April 2025, can continue to use this feature until it is removed in a future release. The Images Assurance board in the web console provides lists for scanned images and running images. In the left navbar in the web console, click **Image Assurance**, **All Scanned Images**. ## All Scanned Images tab[​](#all-scanned-images-tab) This tab lists scanned images if you have enabled or used one of the [Image Assurance scanners](https://docs.tigera.io/calico-cloud/image-assurance/scanners/overview). To manage your scan results, you can filter, dismiss, or delete them: - **Filter:** Apply different combinations of filters to refine your results, making it easier to focus on relevant vulnerabilities for remediation. - **Dismiss:** Hide specific results from the scan results list. You can view a list of dismissed scan results by using the **Dismissed** filter. - **Delete:** Permanently remove specific results from the scan results list. ## Running Images tab[​](#running-images-tab) The tab lists active container images *for all connected managed clusters*. It provides the CVEs associated with running pods to help you assess pod vulnerability. This tab is disabled by default. To enable Running Images, click the **Setting** icon in the top right corner, then select **Enable Runtime View**. > **SECONDARY:** If you are using the CLI scanner and your cluster does not use the default containerd socket path (`/run/containerd/containerd.sock`), you must change the path to allow the Running Images service to collect image information. To update the CRI socket path for a cluster, run the following command: > > ```bash > kubectl patch imageassurance default --type='merge' -p '{"spec":{"criSocketPath":""}}' > ``` > > For details, see the [Image Assurance installation reference](https://docs.tigera.io/calico-cloud/reference/installation/ia-api#image-assurance.operator.tigera.io/v1.ImageAssuranceSpec). Other notes: - The columns, **Clusters** and **Running Instances**, show the number of running instances in clusters that are connected to Calico Cloud. - The **Unknown** scan result filter reflects images that are not fully scanned. Because they can add noise to the table, they are disabled by default. To enable Unknown results for strategic troubleshooting, click the **Result** drop-down menu and select **Unknown**. - In the All Scanned Images and Running Images tabs, the **Registry path** field may be blank if Calico Cloud cannot access this metadata. For example, images from Docker Hub do not specify the registry in the image metadata. - Any exceptions configured for image scanning will be applicable to Runtime View as well. ## Image assessment: Pass, Warn, and Fail[​](#image-assessment-pass-warn-and-fail) The Image Assurance image assessment is based on the [Common Vulnerability Scoring System v3 (CVSS Scores)](https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator). The following table shows how Image Assurance scores map to CVSS scores. | CVSS v3 scores | Image Assurance mapping | Default settings | | -------------------------------------- | ----------------------- | -------------------- | | 0.0 – (None) 0.1 – 3.9 (Low) | Pass = 3.9 | Low | | 4.0 – 6.9 (Medium) | Warn = 7 | Medium severity – 7 | | 7.0 – 8.9 (High) 9.0 – 10.0 (Critical) | Fail = 8.0 | Critical or high – 8 | CVEs without a CVSS v3 score (too old to have one, or are too new to be assigned one), display a blank score in the UI, and **N/A** in the CLI. ### Changing the default CVSS threshold values[​](#changing-the-default-cvss-threshold-values) The following default threshold values will work for the majority of Calico Cloud deployments. However, you may need to change the defaults because of security requirements. ![scan-settings](https://docs.tigera.io/assets/images/scan-settings-f88ece98681d33bfde22c3b236e40a52.png) To change the CVSS threshold values, note the following: - Changes to threshold values take effect immediately and alter the scan results for images already in the system - If you are using admission controller policies, changing a value may allow pods in your Kubernetes cluster that were previously being blocked, to now be deployed or vice versa. ## Exploit Data[​](#exploit-data) The scanning process also attaches EPSS and known exploit data onto each image and vulnerability viewable through the UI. An EPSS score of 0.1 - or 10% - means a vulnerability has a 10% probability that it will be exploited in the wild within the next 30 days. Users should use this information alongside CVSS scores to prioritize remediating vulnerabilities. For example, you may not have the time to remediate all critical vulnerabilities, but you can use the EPSS score to help prioritize. By additionally filtering with an EPSS score of > 90% you can target the critical vulnerabilities that are most likely to be exploited. Note that an EPSS score of > 10% can be considered a high number. Information about [The EPSS Model](https://www.first.org/epss/model) can be found on the EPSS website created by [FIRST](https://www.first.org/). Known exploits are based off of the [CISA KEV Catalog](https://www.cisa.gov/known-exploited-vulnerabilities-catalog), a list of vulnerabilities that have been exploited in the wild and maintained by CISA. ## Export results[​](#export-results) From each tab, you can export data or a JSON file with image URLs. Exporting data is based on the images in the list and the current filter selections. CSV table options include: - **Export one row per image** - export one row for each image with all associated CVEs condensed into a single column. - **Export one row for each image and CVE ID** - export a unique image plus CVE combination for each row. For example, if an image has 10 CVEs, 10 rows are created (1 for each CVE). > **SECONDARY:** Images without associated CVEs are not included in the exported data (regardless if they are included by filters). ## Next steps[​](#next-steps) - [Set up alerts on vulnerabilities](https://docs.tigera.io/calico-cloud/image-assurance/set-up-alerts) ### Exclude vulnerabilities from scan results > **WARNING:** This feature was deprecated in Calico Cloud version 21.1.0 and will be removed in a future release. Availability depends on when you started using Calico Cloud. > > - For users who started using Calico Cloud in April 2025 or later, this feature is not available. > - Legacy users, who started using Calico Cloud before April 2025, can continue to use this feature until it is removed in a future release. This guide shows you how to exclude vulnerabilities from your image scan results. By excluding vulnerabilities that don't require action, such as false positives, you can reduce the volume of reported vulnerabilities that you need to deal with. You can set the exclusion for a particular vulnerability to apply to a particular image version, all images in a repository, or all images in the system. > **WARNING:** Excluding vulnerabilities from image results can significantly change how Image Assurance determines image health. You may need to take corrective action to stabilize your workflow. > > - **Maximum CVSS score:** An image’s maximum CVSS score may be reduced to a lower score. An exception could eliminate the vulnerability with the highest CVSS score for an image. > - **Scan assessment:** The assessment value (Pass, Warn, or Fail) could change because it is based on the maximum CVSS score. For example, Fail could change to Warn, and Warn could change to Pass. > - **Vulnerability global alerts:** Alerts may no longer be triggered. Alerts are triggered based on scan results or maximum CVSS score values of images. > - **Admission controller policies:** Pods could be created where they were previously blocked. Admission controller policies are based on vulnerability information (scan result or maximum CVSS score). ## Exclude a vulnerability from your scan results[​](#exclude-a-vulnerability-from-your-scan-results) You can exclude a vulnerability from your scan results page. 1. From the web console, go to **Image Assurance > All Scan Results**. 2. Select an image from the list. 3. On the scan results panel that appears, expand a package, and then click **Actions > Add Exception**. 4. On the **Add exception** dialog, select a scope, enter an explanation for why you're creating an exception, and then click **Save**. ## Exclude multiple vulnerabilities at the same time[​](#exclude-multiple-vulnerabilities-at-the-same-time) To exclude large numbers of vulnerabilities, you can organize the required information as a CSV file and import it directly to Calico Cloud. 1. Optional: You can start with a preformatted list of all vulnerabilities in your cluster, and then edit the list to include only those vulnerabilities you want to exclude. 2. From the web console, go to **Image Assurance > All Scan Results**. 3. On the **All Scan Results** page, click the **Export** button, and then click **Export data**. 4. On the **Export** dialog that appears, select the **CSV** data type and the **Export a row for each image and CVE IDs** option for the table style. Click **Export** to download the CSV file. 5. Edit the CSV file 6. Prepare the CSV file with information about the vulnerabilities you want to exclude. The file must contain the following information: | Column header | Description | Example | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | | `CVE` | The CVE identifier. Required by the `any`, `repo`, and `image` scopes. | `CVE-2024-1234` | | `Registry` | The URL path to the container registry. Required by the `repo` and `image` scopes. | `mycontainerregistry.io/my-org` | | `Repository` | The container image name. Required by the `repo` and `image` scopes. | `my-application` | | `Tags` | A JSON list of image tags. Required by the `image` scope. | `"[""v1.2.3"",""v2.3.4""]"` | | `Justification` | Your explanation for why you're excluding this vulnerability. Required by the `any`, `image`, and `image` scopes. | `This one is a false positive.` | | `Scope` | Determines whether the vulnerability exception applies only to specific tagged images, to any image in a specific repository, or to any image where the vulnerability is found. One of the following values is required: • `any`: The exception applies to all images. • `repo`: The exception applies to all versions of an image in a repository. • `image`: The exception applies to a specific, tagged version of an image. | `image` | > **SUCCESS:** An exported vulnerability list (see step 1) includes many more columns than what is required to import vulnerability exceptions in bulk. You do not need to remove or reorganize the extraneous information, but you do need to add two new columns for `Justification` and `Scope`. Example: a CSV list of vulnerability exclusion definitions ```title CVE,Registry,Repository,Tags,Justification,Scope CVE-2024-1234,mycontainerregistry.io/my-org,my-application,"[""v3.4.5""]",justification,image CVE-2024-2345,mycontainerregistry.io/my-org,my-application,"[""v1.2.3"",""v2.3.4""]",justification,image CVE-2024-3456,mycontainerregistry.io/my-org,my-application,,justification,repo CVE-2024-4567,,,,justification,any ``` 1. In the web console, go to **Image Assurance > Vulnerability Exceptions**, and then click  **Upload exceptions**. 2. Select the CSV file you created, and then click **Upload file**. After the data is validated, you'll see a summary of the exceptions. If there are errors, modify your CSV file and repeats steps 1 and 2. 3. Review the processed exceptions on the summary page click **Create exceptions**. The new exceptions are listed on the **Vulnerability Exceptions** page. ### Set up alerts on vulnerabilities > **WARNING:** This feature was deprecated in Calico Cloud version 21.1.0 and will be removed in a future release. Availability depends on when you started using Calico Cloud. > > - For users who started using Calico Cloud in April 2025 or later, this feature is not available. > - Legacy users, who started using Calico Cloud before April 2025, can continue to use this feature until it is removed in a future release. ## Big picture[​](#big-picture) Create alerts on high-severity vulnerabilities so you can delegate remediation efforts to the appropriate team. ## How to[​](#how-to) To create alerts, use the [Global alert resource](https://docs.tigera.io/calico-cloud/reference/resources/globalalert). ### Example 1 - Alert on a failed image[​](#example-1---alert-on-a-failed-image) In this example, an alert is created whenever there is more than one event for an image from the specified registry/repo that has a result value of Fail within the past 30 minutes. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalAlert metadata: name: example-1 spec: summary: 'Vulnerabilities for a specific repo based on results' description: 'Vulnerabilities for a specific repo based on results' severity: 100 period: 30m lookback: 30m dataSet: vulnerability query: registry="quay.io/tigera" AND repository="node" AND result="Fail" metric: count condition: gt threshold: 1 ``` ### Example 2 - Alert on a specific repo with maximum CVSS greater than 7.0[​](#example-2---alert-on-a-specific-repo-with-maximum-cvss-greater-than-70) In this example, an alert is created whenever there is at least one event for an image from the specified registry/ repo that has a max CVSS score greater than 7.0 within the past 30 minutes. Providing control over the exact max CVSS score threshold lets you define a trigger that is different from what the CVSS score threshold is configured for Fail scan result on the Scan Results page in the web console. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalAlert metadata: name: example-2 spec: summary: 'Vulnerabilities for a specific repo based on max CVSS score' description: 'Vulnerabilities for a specific repo based on max CVSS score' severity: 100 period: 30m lookback: 30m dataSet: vulnerability query: registry="quay.io/tigera" AND repository="node" field: max_cvss_score metric: max condition: gt threshold: 7.0 ``` ### Example 3 - Alert on a failed scan result within a pod in a namespace[​](#example-3---alert-on-a-failed-scan-result-within-a-pod-in-a-namespace) In this example, an alert is created whenever there is at least one event for an image that has a scan result of Fail, that is running within a pod in any cluster in the namespace `tigera-elasticsearch`, within the past 30 minutes. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalAlert metadata: name: example-3 spec: summary: 'Vulnerabilities for namespace tigera-elasticsearch' description: 'Vulnerabilities for namespace tigera-elasticsearch' severity: 100 period: 30m lookback: 30m dataSet: vulnerability query: result="Fail" AND namespace="tigera-elasticsearch" metric: count condition: gt threshold: 1 ``` > **SECONDARY:** You must apply global alerts across all applicable managed clusters. For example, if you have five clusters, you must apply the alerts five times. For a complete list of parameters, see [Global alert resource](https://docs.tigera.io/calico-cloud/reference/resources/globalalert). ### Create policy to block vulnerable images from your cluster > **WARNING:** This feature was deprecated in Calico Cloud version 21.1.0 and will be removed in a future release. Availability depends on when you started using Calico Cloud. > > - For users who started using Calico Cloud in April 2025 or later, this feature is not available. > - Legacy users, who started using Calico Cloud before April 2025, can continue to use this feature until it is removed in a future release. > **SECONDARY:** This feature is tech preview. Tech preview features may be subject to significant changes before they become GA. ## Big picture[​](#big-picture) Protecting your cluster from vulnerable images can be very difficult. An image that appears to be secure today could contain a newly-discovered vulnerability tomorrow, and acting on this new information in real time can be challenging. Calico Cloud’s Image Assurance Admission Controller automatically blocks resources that would create containers with vulnerable images from entering your cluster using the latest vulnerability data and scan results. ## Concepts[​](#concepts) ### About the Calico Cloud Admission Controller[​](#about-the-calico-cloud-admission-controller) Calico Cloud uses [Kubernetes Validating Webhook Configuration](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/) to register the Calico Cloud Admission Controller as a callback to accept or reject resources that create pods (such as deployments and daemonsets). ### How the Admission Controller evaluates admission requests[​](#how-the-admission-controller-evaluates-admission-requests) The Calico Cloud Admission Controller blocks the creation or modification of resources that create pods. If a resource that creates pods is admitted by the Admission Controller, *the pods it creates are not reevaluated*. For example, if you create a deployment, the Admission Controller receives an admission request and either allows or rejects the request. If allowed, the Admission Controller will not evaluate the request for the replica set that Kubernetes creates for the deployment or the pod that is created for the replica set. Why? Because it could destabilize a production cluster if new vulnerabilities are found for deployed images and pods are restarted. However, if you *create a pod directly*, the Admission Controller evaluates the admission request for the pod and allows or rejects the request. ### About container admission policies[​](#about-container-admission-policies) Container admission policies are custom Kubernetes resources that allow you to configure the criteria for the Admission Controller to reject admission requests for resources that create pods. ## How to[​](#how-to) - [Install the Admission Controller](#install-the-admission-controller) - [Create container admission policies](#create-container-admission-policies) - [Troubleshoot](#troubleshoot) ### Install the Admission Controller[​](#install-the-admission-controller) 1. Create a directory to download the manifests/scripts needed for the installation. ```bash mkdir admission-controller-install && cd admission-controller-install ``` 2. Generate the certificate key pair. For secure TLS communication between the Kubernetes API server and the Admission controller, you must generate a TLS certificate and key pair. You can either generate the TLS key and certificate yourself and move them to the current folder under the names `admission_controller_key.pem` and `admission_controller_cert.pem`, or use the following command to generate the pair: ```bash export URL="https://installer.calicocloud.io/manifests/v3.22.0-2.0-7/manifests" && curl ${URL}/generate-open-ssl-key-cert-pair.sh | bash ``` > **WARNING:** If you generate the key and certificate pair yourself, you must set the SANS to `tigera-image-assurance-admission-controller-service.tigera-image-assurance.svc`. 3. Download and configure the Admission Controller manifests. As a safety mechanism, we require that you specify the namespaces that the Admission Controller will apply in the `IN_NAMESPACE_SELECTOR_KEY` and the `IN_NAMESPACE_SELECTOR_VALUES`. These values configure the Kubernetes API server to send admission requests to our Admission Controller only for resources from relevant namespaces. For example, to configure the Kubernetes API server to send admission requests for resources created in any namespace with label key `name`, and label values either `prod` or `staging-test`, set the variables as follows: ```bash export IN_NAMESPACE_SELECTOR_KEY="name" && export IN_NAMESPACE_SELECTOR_VALUES="prod staging-test" ``` Here is the namespace manifest with the staging-test label and name key. ```yaml kind: Namespace apiVersion: v1 metadata: name: staging-test labels: name: staging-test ``` > **WARNING:** Do not add Kubernetes critical namespaces such as the `kube-system` namespace. This could create a deadlock situation where you cannot bring up the Admission Controller because a critical Kubernetes pod is not running, but you also cannot bring up the critical Kubernetes pod because the Admission Controller is not running. ```bash export IN_NAMESPACE_SELECTOR_KEY="name" && \ export IN_NAMESPACE_SELECTOR_VALUES="prod staging-test" && \ curl ${URL}/install-ia-admission-controller.sh | bash ``` 4. Apply the Admission Controller manifests. ```bash kubectl apply -f ./tigera-image-assurance-admission-controller-deploy.yaml ``` ## Create container admission policies[​](#create-container-admission-policies) Container admission policies are used to define criteria for the Admission Controller to admit or reject admission for resources that create pods. For details, see [ContainerAdmissionPolicies](https://docs.tigera.io/calico-cloud/reference/resources/containeradmissionpolicy). **Sample container admission policies** This ContainerAdmissionPolicy allows admission requests for pod-creating resources whose image is in the registry/repository `gcr.io/company/production-repository/*`, with a scan status of either `Pass` or `Warn`, and rejects all other admission requests. ```yaml apiVersion: containersecurity.tigera.io/v1beta1 kind: ContainerAdmissionPolicy metadata: name: reject-failed-and-non-gcr spec: selector: all() namespaceSelector: all() order: 10 rules: - action: 'Reject' imagePath: operator: IsNoneOf values: - '^gcr.io/company/production-repository/.*' - action: Allow imageScanStatus: operator: IsOneOf values: - Pass - Warn - action: Reject ``` The following ContainerAdmissionPolicy rejects deploying or updating pod-creating resources with the label, `reject-policy: reject-outdated-scans`, from any namespace labeled, `apply-container-admission-policies == 'true'`, that would deploy an image that hasn't been scanned in 3 days. ```yaml apiVersion: containersecurity.tigera.io/v1beta1 kind: ContainerAdmissionPolicy metadata: name: reject-failed-and-non-gcr spec: selector: "reject-policy == 'reject-outdated-scans'" namespaceSelector: "apply-container-policies == 'true'" order: 1 rules: - action: Allow imageLastScan: operator: 'gt' duration: days: 3 - action: Reject ``` The first rule (Allow), allows images based on the age of the image scan (in days). In this example, we want to allow images that have been scanned within the last three days. So, we use the gt operator (greater than), along with Duration, 3 days, to say the image scan time must be greater than 3 days ago but less than now. "Now" is defined as when the attempt was made to create the Kubernetes resource. If the Allow rule does not match, then the second action rule (Reject) is evaluated, which denies everything. You can also use modify the Allow rule to match an absolute time. For help, see [ContainerAdmissionPolicies](https://docs.tigera.io/calico-cloud/reference/resources/containeradmissionpolicy). ## Troubleshoot[​](#troubleshoot) **My container admission policy is not blocking resources from a namespace, even though the namespaceSelector matches the namespace** This indicates that the Kubernetes API server is not sending admission requests for the namespace. Verify that the key and value(s) that you specified for `IN_NAMESPACE_SELECT_KEY` and `IN_NAMESPACE_SELECT_VALUES` in the installation steps, match the policy namespaces. ### Create a Jira issue for your image scan results > **WARNING:** This feature was deprecated in Calico Cloud version 21.1.0 and will be removed in a future release. Availability depends on when you started using Calico Cloud. > > - For users who started using Calico Cloud in April 2025 or later, this feature is not available. > - Legacy users, who started using Calico Cloud before April 2025, can continue to use this feature until it is removed in a future release. You can create and assign Jira issues with information about vulnerabilties from your Image Assurance scan results. ![ia-jira-after-creation](https://docs.tigera.io/assets/images/ia-jira-after-creation-92e2960f74edcc9c69005ac998bf5713.png) *Scan results detail with link to Jira issue.* ## Add Jira credentials to Calico Cloud[​](#add-jira-credentials-to-calico-cloud) You must add Jira user credentials to Calico Cloud to create issues for scanned images. ***Prerequisites*** - You have access to a Jira user account with permissions to create issues in a project. - For the Jira user account, you have: - An Atlassian site URL. If you access Jira at the URL `https://.atlassian.net/jira`, then your site URL is `.atlassian.net`. - An API token. For details on how to get an API token, see [Manage API tokens for your Atlassian account](https://support.atlassian.com/atlassian-account/docs/manage-api-tokens-for-your-atlassian-account/). 1. In the web console, the user icon **> Settings**. 2. Select the **Jira** tab, complete the fields with information about your Jira user account, and then click **Save**. 3. Select the Jira project you want Calico Cloud to create issues for, and then click **Save**. ## Create a Jira issue for a scanned image[​](#create-a-jira-issue-for-a-scanned-image) You can create and assign a Jira issue directly from the scan results information page for an image. 1. From the web console, click **Image Assurance >All Scanned Images**. 2. Click an item in the list of scanned images to open a detailed view of the vulnerabilities in that image. 3. In the **JIRA** section, click **Add Ticket**. 4. In the **Add Jira issue** dialog, complete the fields and click **Create Jira issue**. A link to the new Jira issue will be added to the detailed view page. --- ## Compliance and security ### Compliance and security Get reports on Kubernetes workloads and environments for regulatory compliance. Encrypt traffic in your cluster with WireGuard. ##### [Enable compliance reports](https://docs.tigera.io/calico-cloud/compliance/enable-compliance) [Enable compliance reports to configure reports to assess compliance for all assets in a Kubernetes cluster.](https://docs.tigera.io/calico-cloud/compliance/enable-compliance) ##### [Schedule and run compliance reports](https://docs.tigera.io/calico-cloud/compliance/overview) [Get the reports for regulatory compliance on Kubernetes workloads and environments.](https://docs.tigera.io/calico-cloud/compliance/overview) ##### [Configure CIS benchmark reports](https://docs.tigera.io/calico-cloud/compliance/compliance-reports-cis) [Configure reports to assess compliance for all assets in a Kubernetes cluster.](https://docs.tigera.io/calico-cloud/compliance/compliance-reports-cis) ##### [Encrypt data in transit](https://docs.tigera.io/calico-cloud/compliance/encrypt-cluster-pod-traffic) [Enable WireGuard for state-of-the-art cryptographic security between pods for Calico Enterprise clusters.](https://docs.tigera.io/calico-cloud/compliance/encrypt-cluster-pod-traffic) ##### [Configure an outbound HTTP proxy](https://docs.tigera.io/calico-cloud/compliance/configure-http-proxy) [Configure an HTTP proxy to use for connections that leave the cluster](https://docs.tigera.io/calico-cloud/compliance/configure-http-proxy) ### Enable compliance reports > **WARNING:** Compliance reports are deprecated and will be removed in a future release. We're building a new compliance reporting system that will eventually replace the current one. ## Big picture[​](#big-picture) Enabling compliance reports improves the cluster's compliance posture. It involves generating compliance reports for Kubernetes clusters based on archived flow and audit logs for Calico Enterprise and Kubernetes resources. The process includes components for snapshotting configurations, generating reports, managing jobs, providing APIs with RBAC, and benchmarking security. ## Value[​](#value) The compliance system consists of several key components that work together to ensure comprehensive compliance monitoring and reporting: - `compliance-snapshotter` : Lists required configurations and pushes snapshots to Elasticsearch, providing visibility into configuration changes. - `compliance-reporter` : Generates reports by analyzing configuration history, determining configuration evolution and identifying "worst-case outliers." - `compliance-controller` : Manages the creation, deletion, and monitoring of report generation jobs. - `compliance-server` : Offers API for report management and enforces RBAC. - `compliance-benchmarker` : Runs CIS Kubernetes Benchmark checks on each node to ensure secure deployment. ### Enable compliance reports using kubectl[​](#enable-compliance-reports-using-kubectl) - Create a compliance custom resource, named `tigera-secure`, in the cluster. ```bash kubectl apply -f - < **WARNING:** Compliance reports are deprecated and will be removed in a future release. We're building a new compliance reporting system that will eventually replace the current one. ## Big picture[​](#big-picture) Schedule and run compliance reports to assess Kubernetes workloads and environments for regulatory compliance. ## Value[​](#value) Compliance tools that rely on periodic snapshots, do not provide accurate assessments of Kubernetes workloads against your compliance standards. Calico Cloud compliance dashboard and reports provide a complete inventory of regulated workloads, along with evidence of enforcement of network controls for these workloads. Additionally, audit reports are available to see changes to any network security controls. ## Concepts[​](#concepts) ### Compliance reports at a glance[​](#compliance-reports-at-a-glance) Compliance report are based on archived flow logs and audit logs for all of your Calico Cloud resources, plus any audit logs you've configured for Kubernetes resources in the Kubernetes API server: - Pods - Host endpoints - Service accounts - Namespaces - Kubernetes service endpoints - Global network sets - Calico and Kubernetes network policies - Global network policies Compliance reports provide the following high-level information: - **Protection** - Endpoints explicitly protected using ingress or egress policy - Endpoints with Envoy enabled - **Policies and services** - Policies and services associated with endpoints - Policy audit logs - **Traffic** - Allowed ingress/egress traffic to/from namespaces - Allowed ingress/egress traffic to/from the internet ![compliance-reporting](https://docs.tigera.io/assets/images/compliance-reporting-cd3a4b948179bc1af1a33d29b20e7897.png) ## Before you begin[​](#before-you-begin) **Unsupported** - AKS - GKE - OpenShift - TKG **Required** - You [Enabled compliance reports](https://docs.tigera.io/calico-cloud/compliance/enable-compliance) * Ensure that all nodes in your Kubernetes clusters are time-synchronized using NTP or similar (for accurate audit log timestamps) * [Configure audit logs for Kubernetes resources](https://docs.tigera.io/calico-cloud/observability/elastic/audit-overview) You must configure audit logs for Kubernetes resources through the Kubernetes API to get a complete view of all resources. ## How to[​](#how-to) - [Configure report permissions](#configure-report-permissions) - [Configure and schedule reports](#configure-and-schedule-reports) - [View report generation status](#view-report-generation-status) - [Run reports](#run-reports) ### Configure report permissions[​](#configure-report-permissions) Report permissions are granted using the standard Kubernetes RBAC based on ClusterRole and ClusterRoleBindings. The following table outlines the required RBAC verbs for each resource type for a specific user actions. | **Action** | **globalreporttypes** | **globalreports** | **globalreports/status** | | ------------------------------------------------------- | ------------------------------- | --------------------------------- | ------------------------ | | Manage reports (create/modify/delete) | | \* | get | | View status of report generation through kubectl | | get | get | | List the generated reports and summary status in the UI | | list + get (for required reports) | | | Export the generated reports from the UI | get (for the particular report) | get (for required reports) | | The following sample manifest creates RBAC for three users: Paul, Candice and David. - Paul has permissions to create/modify/delete the report schedules and configuration, but does not have permission to export generated reports from the UI. - Candice has permissions to list and export generated reports from the UI, but cannot modify the report schedule or configuration. - David has permissions to list and export generated `dev-inventory` reports from the UI, but cannot list or download other reports, nor modify the report schedule or configuration. ```yaml kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: tigera-compliance-manage-report-config rules: - apiGroups: ['projectcalico.org'] resources: ['globalreports'] verbs: ['*'] - apiGroups: ['projectcalico.org'] resources: ['globalreports/status'] verbs: ['get', 'list', 'watch'] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: tigera-compliance-manage-report-config subjects: - kind: User name: paul apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: tigera-compliance-manage-report-config apiGroup: rbac.authorization.k8s.io --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: tigera-compliance-list-download-all-reports rules: - apiGroups: ['projectcalico.org'] resources: ['globalreports'] verbs: ['get', 'list'] - apiGroups: ['projectcalico.org'] resources: ['globalreporttypes'] verbs: ['get'] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: tigera-compliance-list-download-all-reports subjects: - kind: User name: candice apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: tigera-compliance-list-download-all-reports apiGroup: rbac.authorization.k8s.io --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: tigera-compliance-list-download-dev-inventory rules: - apiGroups: ['projectcalico.org'] resources: ['globalreports'] verbs: ['list'] - apiGroups: ['projectcalico.org'] resources: ['globalreports'] verbs: ['get'] resourceNames: ['dev-inventory'] - apiGroups: ['projectcalico.org'] resources: ['globalreporttypes'] verbs: ['get'] resourceNames: ['dev-inventory'] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: tigera-compliance-list-download-dev-inventory subjects: - kind: User name: david apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: tigera-compliance-list-download-dev-inventory apiGroup: rbac.authorization.k8s.io ``` ### Configure and schedule reports[​](#configure-and-schedule-reports) To configure and schedule a compliance report, create a [GlobalReport](https://docs.tigera.io/calico-cloud/reference/resources/globalreport) with the following information. | **Fields** | **Description** | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | name | Unique name for your report. | | reportType | One of the following predefined report types: `inventory`, `network-access`, `policy-audit`. | | schedule | The start and end time of the report using [crontab format](https://en.wikipedia.org/wiki/Cron). To allow for archiving, reports are generated approximately 30 minutes after the end time. A single report is limited to a maximum of two per hour. | | endpoints | **Optional**. For inventory and network-access reports, specifies the endpoints to include in the report. For the policy-audit report, restricts audit logs to include only policies that apply to the selected endpoints. If not specified, the report includes all endpoints and audit logs. | | jobNodeSelector | **Optional**. Limits report generation jobs to specific nodes. | | suspend | **Optional**. Suspends report generation. All in-flight reports will complete, and future scheduled reports are suspended. | > **SECONDARY:** GlobalReports can only be configured using kubectl (not calicoctl); and they cannot be edited in the Tigera Secure EE the web console. The following sections provide sample schedules for the predefined reports. ### Weekly reports, all endpoints[​](#weekly-reports-all-endpoints) The following report schedules weekly inventory reports for *all* endpoints. The jobs that create the reports will run on the infrastructure nodes (e.g. nodetype == 'infrastructure'). ```yaml apiVersion: projectcalico.org/v3 kind: GlobalReport metadata: name: weekly-full-inventory spec: reportType: inventory schedule: 0 0 * * 0 jobNodeSelector: nodetype: infrastructure ``` ### Daily reports, selected endpoints[​](#daily-reports-selected-endpoints) The following report schedules daily inventory reports for production endpoints (e.g. deployment == ‘production’). ```yaml apiVersion: projectcalico.org/v3 kind: GlobalReport metadata: name: daily-production-inventory spec: reportType: inventory endpoints: selector: deployment == 'production' schedule: 0 0 * * * ``` ### Hourly reports, endpoints in named namespaces[​](#hourly-reports-endpoints-in-named-namespaces) The following report schedules hourly network-access reports for the accounts department endpoints, that are specified using the namespace names: **payable**, **collections** and **payroll**. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalReport metadata: name: hourly-accounts-networkaccess spec: reportType: network-access endpoints: namespaces: names: ['payable', 'collections', 'payroll'] schedule: 0 * * * * ``` ### Daily reports, endpoints in selected namespaces[​](#daily-reports-endpoints-in-selected-namespaces) The following report schedules daily network-access reports for the accounts department with endpoints specified using a namespace selector. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalReport metadata: name: daily-accounts-networkaccess spec: reportType: network-access endpoints: namespaces: selector: department == 'accounts' schedule: 0 0 * * * ``` ### Monthly reports, endpoints for named service accounts in named namespaces[​](#monthly-reports-endpoints-for-named-service-accounts-in-named-namespaces) The following schedules monthly audit reports. The audited policy is restricted to policy that applies to widgets/controller endpoints specified by the namespace **widgets** and service account **controller**. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalReport metadata: name: monthly-widgets-controller-tigera-policy-audit spec: reportType: policy-audit schedule: 0 0 1 * * endpoints: serviceAccounts: names: ['controller'] namespaces: names: ['widgets'] ``` ### View report generation status[​](#view-report-generation-status) To view the status of a report, you must use the `kubectl` command. For example: ```bash kubectl get globalreports.projectcalico.org daily-inventory.p -o yaml ``` In a report, the job status types are: - **lastScheduledReportJob**: The most recently scheduled job for generating the report. Because reports are scheduled in order, the “end time” of this report will be the “start time” of the next scheduled report. - **activeReportJobs**: Default = allows up to 5 concurrent report generation jobs. - **lastFailedReportJobs**: Default = keeps the 3 most recent failed jobs and deletes older ones. A single report generation job will be retried up to 6 times (by default) before it is marked as failed. - **lastSuccessfulReportJobs**: Default = keeps the 2 most recent successful jobs and deletes older ones. ### Change the default report generation time[​](#change-the-default-report-generation-time) By default, reports are generated 30 minutes after the end of the report, to ensure all of the audit data is archived. (However, this gap does not affect the data collected “start/end time” for a report.) You can adjust the time for audit data for cases like initial report testing, to demo a report, or when manually creating a report that is not counted in global report status. To change the delay, go to the installation manifest, and uncomment and set the environment `TIGERA_COMPLIANCE_JOB_START_DELAY`. Specify value as a [Duration string](https://golang.org/pkg/time/#ParseDuration). ### Run reports[​](#run-reports) You can run reports at any time to specify a different start/end time, and if a scheduled report fails. Calico Cloud GlobalReport schedules Kubernetes Jobs, which create a single-run pod to generate a report and store it in Elasticsearch. Because you need to run reports as a pod, you need higher permissions: allow `create` access access for pods in namespace `tigera-compliance` using the `tigera-compliance-reporter` service account. To run a report on demand: 1. Download the pod template corresponding to your installation method. ```bash curl https://downloads.tigera.io/ee/v3.22.0-2.0/manifests/compliance-reporter-pod-managed.yaml -o compliance-reporter-pod.yaml ``` 2. Edit the template as follows: - Edit the pod name if required. - If you are using your own docker repository, update the container image name with your repo and image tag. - Set the following environments according to the instructions in the downloaded manifest: - `TIGERA_COMPLIANCE_REPORT_NAME` - `TIGERA_COMPLIANCE_REPORT_START_TIME` - `TIGERA_COMPLIANCE_REPORT_END_TIME` 3. Apply the updated manifest, and query the status of the pod to ensure it completes. Upon completion, the report is available in the Calico Cloud web console. ```bash # Apply the compliance report pod kubectl apply -f compliance-reporter-pod.yaml # Query the status of the pod kubectl get pod -n tigera-compliance ``` > **SECONDARY:** Manually-generated reports do not appear in GlobalReport status. ## Additional resources[​](#additional-resources) - For details on configuring and scheduling reports, see [Global reports](https://docs.tigera.io/calico-cloud/reference/resources/globalreport) - For report field descriptions, see [Compliance reports](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/) - [CIS benchmarks](https://docs.tigera.io/calico-cloud/compliance/compliance-reports-cis) ### Configure CIS benchmark reports > **WARNING:** Compliance reports are deprecated and will be removed in a future release. We're building a new compliance reporting system that will eventually replace the current one. ## Big picture[​](#big-picture) Use the Calico Cloud Kubernetes CIS benchmark report to assess compliance for all assets in a Kubernetes cluster. ## Value[​](#value) A standard requirement for an organization’s security and compliance posture is to assess your Kubernetes clusters against CIS benchmarks. The Calico Cloud Kubernetes CIS benchmark report provides this comprehensive view into your Kubernetes clusters while strengthening your threat detection capability by looking beyond networking data. ## Concepts[​](#concepts) ### Default settings and configuration[​](#default-settings-and-configuration) During Calico Cloud installation, each node starts a pod named, `compliance-benchmarker`. A preconfigured Kubernetes CIS benchmark report is generated every hour. You can view the report in **Compliance**, **Compliance Reports**, download it to .csv format. To schedule the CIS benchmark report or change settings, use the **global report** resource. Global reports are configured as YAML files and are applied using `kubectl`. ### Best practices[​](#best-practices) We recommend that you review the CIS benchmark best practices for securing cluster component configurations here: [CIS benchmarks downloads](https://learn.cisecurity.org/benchmarks). ## Before you begin[​](#before-you-begin) **Required** - You [Enabled compliance reports](https://docs.tigera.io/calico-cloud/compliance/enable-compliance) **Limitations** CIS benchmarks runs only on nodes where Calico Cloud is running. This limitation may exclude control plane nodes in some managed cloud platforms (AKS, EKS, GKE). Because the user has limited control over installation of control plane nodes in managed cloud platforms, these reports may have limited use for cloud users. ## How to[​](#how-to) - [Configure and schedule CIS benchmark reports](#configure-and-schedule-cis-benchmark-reports) - [View report generation status](#view-report-generation-status) - [Review and address CIS benchmark results](#review-and-address-cis-benchmark-results) - [Manually run reports](#manually-run-reports) - [Troubleshooting](#troubleshooting) ### Configure and schedule CIS benchmark reports[​](#configure-and-schedule-cis-benchmark-reports) Verify that the `compliance-benchmarker` is running and the `cis-benchmark` report type is installed. ```bash kubectl get -n tigera-compliance daemonset compliance-benchmarker kubectl get globalreporttype cis-benchmark ``` In the following example, we use a **GlobalReport** with CIS benchmark fields to schedule and filter results. The report is scheduled to run at midnight of the next day (in UTC), and the benchmark items 1.1.4 and 1.2.5 will be omitted from the results. | **Fields** | **Description** | | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | schedule | The start and end time of the report using [crontab format](https://en.wikipedia.org/wiki/Cron). To allow for archiving, reports are generated approximately 30 minutes after the end time. A single report is limited to a maximum of two per hour. | | highThreshold | **Optional**. Integer percentage value that determines the lower limit of passing tests to consider a node as healthy. Default: 100 | | medThreshold | **Optional**. Integer percentage value that determines the lower limit of passing tests to consider a node as unhealthy. Default: 50 | | includeUnscoredTests | **Optional**. Boolean value that when false, applies a filter to exclude tests that are marked as “Unscored” by the CIS benchmark standard. If true, the tests will be included in the report. Default: true | | numFailedTests | **Optional**. Integer value that sets the number of tests to display in the Top-failed Tests section of the CIS benchmark report. Default: 5 | | resultsFilter | **Optional**. An include or exclude filter to apply on the test results that will appear on the report. | ```yaml apiVersion: projectcalico.org/v3 kind: GlobalReport metadata: name: daily-cis-results labels: deployment: production spec: reportType: cis-benchmark schedule: 0 0 * * * cis: highThreshold: 100 medThreshold: 50 includeUnscoredTests: true numFailedTests: 5 resultsFilters: - benchmarkSelection: { kubernetesVersion: '1.13' } exclude: ['1.1.4', '1.2.5'] ``` ### View report generation status[​](#view-report-generation-status) To view the status of a report, you must use the `kubectl` command. For example: ```bash kubectl get globalreports.projectcalico.org daily-cis-results -o yaml ``` In a report, the job status types are: - **lastScheduledReportJob**: The most recently scheduled job for generating the report. Because reports are scheduled in order, the “end time” of this report will be the “start time” of the next scheduled report. - **activeReportJobs**: Default = allows up to 5 concurrent report generation jobs. - **lastFailedReportJobs**: Default = keeps the 3 most recent failed jobs and deletes older ones. A single report generation job will be retried up to 6 times (by default) before it is marked as failed. - **lastSuccessfulReportJobs**: Default = keeps the 2 most recent successful jobs and deletes older ones. #### Change the default report generation time[​](#change-the-default-report-generation-time) By default, reports are generated 30 minutes after the end of the report, to ensure all of the audit data is archived. (However, this gap does not affect the data collected “start/end time” for a report.) You can adjust the time for audit data for cases like initial report testing, to demo a report, or when manually creating a report that is not counted in global report status. To change the delay, go to the installation manifest, and uncomment and set the environment variable `TIGERA_COMPLIANCE_JOB_START_DELAY`. Specify value as a [Duration string](https://golang.org/pkg/time/#ParseDuration). ### Review and address CIS benchmark results[​](#review-and-address-cis-benchmark-results) We recommend the following approach to CIS benchmark reports results: 1. Download the Kubernetes CIS benchmarks and export your full CIS benchmark results in .csv format. 2. In the compliance dashboard, review the "Top-Failed Tests" section to identify which tests are the most problematic. 3. Cross-reference the top-failed tests to identify which nodes are failing that test. 4. Look up those tests in the [Kubernetes benchmark document](https://downloads.cisecurity.org/#/) and follow the remediation steps to resolve the failure. 5. Discuss with your infrastructure and security team if this remediation is viable within your organization. 6. If so, update your nodes with the fix and ensure that the test passes on the next generation of the report. 7. If the fix is not viable but is an acceptable risk to take within the organization, configure the report specification to exclude that test index so that it no longer appears in the report. 8. If the fix is not viable and not an acceptable risk to take on, keep the failing test within the report so that your team is reminded to address the issue as soon as possible. ### Manually run reports[​](#manually-run-reports) You can manually run reports at any time. For example, run a manual report: - To specify a different start/end time - If a scheduled report fails Calico Cloud GlobalReport schedules Kubernetes Jobs which create a single-run pod to generate a report and store it in Elasticsearch. Because you need to run manual reports as a pod, you need higher permissions: allow `create` access for pods in namespace `tigera-compliance` using the `tigera-compliance-reporter` service account. To manually run a report: 1. Download the pod template corresponding to your installation method.\ **Operator** ```bash curl https://downloads.tigera.io/ee/v3.22.0-2.0/manifests/compliance-reporter-pod-managed.yaml -o compliance-reporter-pod.yaml ``` 2. Edit the template as follows: - Edit the pod name if required. - If you are using your own docker repository, update the container image name with your repo and image tag. - Set the following environments according to the instructions in the downloaded manifest: - `TIGERA_COMPLIANCE_REPORT_NAME` - `TIGERA_COMPLIANCE_REPORT_START_TIME` - `TIGERA_COMPLIANCE_REPORT_END_TIME` 3. Apply the updated manifest, and query the status of the pod to ensure it completes. Upon completion, the report is available in the web console. ```bash # Apply the compliance report pod kubectl apply -f compliance-reporter-pod.yaml # Query the status of the pod kubectl get pod -n=tigera-compliance ``` > **SECONDARY:** Manually-generated reports do not appear in GlobalReport status. ### Troubleshooting[​](#troubleshooting) **Problem**: Compliance reports can fail to generate if the `compliance-benchmarker` component cannot find the required `kubelet` or `kubectl` binaries to determine the Kubernetes version running on the cluster. **Solution or workaround**: If a node is running within a container (not running `kubelet` as a binary), make sure the `kubectl` binary is available in the `/usr/bin` directory. ## Additional resources[​](#additional-resources) - For details on configuring and scheduling reports, see [Global reports](https://docs.tigera.io/calico-cloud/reference/resources/globalreport) - For other predefined compliance reports, see [Compliance reports](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/) ### Encrypt data in transit ## Big picture[​](#big-picture) Enable WireGuard to secure on the wire in-cluster pod traffic in a Calico Cloud cluster. ## Value[​](#value) When this feature is enabled, Calico Cloud automatically creates and manages WireGuard tunnels between nodes providing transport-level security for inter-node, in-cluster pod traffic. WireGuard provides [formally verified](https://www.wireguard.com/formal-verification/) secure and [performant tunnels](https://www.wireguard.com/performance/) without any specialized hardware. For a deep dive in to WireGuard implementation, see this [white paper](https://www.wireguard.com/papers/wireguard.pdf). Calico Cloud supports WireGuard encryption for both IPv4 and IPv6 traffic. These can be independently enabled in the FelixConfiguration resource: `wireguardEnabled` enables encrypting IPv4 traffic over an IPv4 underlay network and `wireguardEnabledV6` enables encrypting IPv6 traffic over an IPv6 underlay network. ## Before you begin[​](#before-you-begin) **Terminology** - Inter-node pod traffic: Traffic leaving a pod from one node destined to a pod on another node - Inter-node, host-network traffic: traffic generated by the node itself or a host-networked-pod destined to another node or host-networked-pod - Same-node pod traffic: Traffic between pods on the same node **Supported encryption** - Inter-node pod traffic: IPv4 only - Inter-node, host-network traffic, IPv4/IPv6: supported only on managed clusters deployed on EKS and AKS **Unsupported** - Encrypted same-node pod traffic - GKE - Using your own custom keys to encrypt traffic **Required** - On all nodes in the cluster that you want to participate in Calico Cloud encryption, verify that the operating system(s) on the nodes are [installed with WireGuard](https://www.wireguard.com/install/). > **SECONDARY:** Some node operating systems do not support WireGuard, or do not have it installed by default. Enabling Calico Cloud WireGuard encryption does not require all nodes to be installed with WireGuard. However, traffic to or from a node that does not have WireGuard installed, will not be encrypted. - IP addresses for every node in the cluster. This is required to establish secure tunnels between the nodes. Calico Cloud can automatically do this using [IP autodetection methods](https://docs.tigera.io/calico-cloud/networking/ipam/ip-autodetection). ## How to[​](#how-to) - [Install WireGuard](#install-wireguard) - [Enable WireGuard for a cluster](#enable-wireguard-for-a-cluster) - [Verify encryption is enabled](#verify-encryption-is-enabled) - [Disable WireGuard for an individual node](#disable-wireguard-for-an-individual-node) - [Disable WireGuard for a cluster](#disable-wireguard-for-a-cluster) ### Install WireGuard[​](#install-wireguard) WireGuard is included in Linux 5.6+ kernels, and has been backported to earlier Linux kernels in some Linux distributions. Install WireGuard on cluster nodes using [instructions for your operating system](https://www.wireguard.com/install/). Note that you may need to reboot your nodes after installing WireGuard to make the kernel modules available on your system. Use the following instructions for these platforms that are not listed on the WireGuard installation page, before proceeding to [enabling WireGuard](#enable-wireguard-for-a-cluster). **Tab: EKS** To install WireGuard on the default Amazon Machine Image (AMI): ```bash sudo yum install kernel-devel-`uname -r` -y sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm -y sudo curl -o /etc/yum.repos.d/jdoss-wireguard-epel-7.repo https://copr.fedorainfracloud.org/coprs/jdoss/wireguard/repo/epel-7/jdoss-wireguard-epel-7.repo sudo yum install wireguard-dkms wireguard-tools -y ``` **Tab: AKS** AKS cluster nodes run Ubuntu with a kernel that has WireGuard installed already, so there is no manual installation required. **Tab: OpenShift** To install WireGuard for OpenShift v4.8: 1. Install requirements: - [CoreOS Butane](https://coreos.github.io/butane/getting-started/) - [Openshift CLI](https://docs.openshift.com/container-platform/4.2/cli_reference/openshift_cli/getting-started-cli.html) 2. Download and configure the tools needed for kmods. ```bash FAKEROOT=$(mktemp -d) git clone https://github.com/tigera/kmods-via-containers cd kmods-via-containers make install FAKEROOT=${FAKEROOT} cd .. git clone https://github.com/tigera/kvc-wireguard-kmod cd kvc-wireguard-kmod make install FAKEROOT=${FAKEROOT} cd .. ``` 1. Configure/edit `${FAKEROOT}/root/etc/kvc/wireguard-kmod.conf`. a. You must then set the URLs for the `KERNEL_CORE_RPM`, `KERNEL_DEVEL_RPM` and `KERNEL_MODULES_RPM` packages in the conf file `$FAKEROOT/etc/kvc/wireguard-kmod.conf`. Obtain copies for `kernel-core`, `kernel-devel`, and `kernel-modules` rpms from [RedHat Access](https://access.redhat.com/downloads/content/package-browser) and host it in an http file server that is reachable by your OCP workers. b. For help configuring `kvc-wireguard-kmod/wireguard-kmod.conf` and WireGuard version to kernel version compatibility, see the [kvc-wireguard-kmod README file](https://github.com/tigera/kvc-wireguard-kmod#quick-config-variables-guide). 2. Get RHEL Entitlement data from your own RHEL8 system from a host in your cluster. ```bash tar -czf subs.tar.gz /etc/pki/entitlement/ /etc/rhsm/ /etc/yum.repos.d/redhat.repo ``` 3. Copy the `subs.tar.gz` file to your workspace and then extract the contents using the following command. ```bash tar -x -C ${FAKEROOT}/root -f subs.tar.gz ``` 4. Transpile your machine config using [CoreOS Butane](https://coreos.github.io/butane/getting-started/). ```bash cd kvc-wireguard-kmod make ignition FAKEROOT=${FAKEROOT} > mc-wg.yaml ``` 5. With the KUBECONFIG set for your cluster, run the following command to apply the MachineConfig which will install WireGuard across your cluster. ```bash oc create -f mc-wg.yaml ``` ### Enable WireGuard for a cluster[​](#enable-wireguard-for-a-cluster) Enable IPv4 WireGuard encryption across all the nodes using the following command. ```bash kubectl patch felixconfiguration default --type='merge' -p '{"spec":{"wireguardEnabled":true}}' ``` Enable IPv6 WireGuard encryption across all the nodes using the following command. ```bash kubectl patch felixconfiguration default --type='merge' -p '{"spec":{"wireguardEnabledV6":true}}' ``` To enable both IPv4 and IPv6 WireGuard encryption across all the nodes, use the following command. ```bash kubectl patch felixconfiguration default --type='merge' -p '{"spec":{"wireguardEnabled":true,"wireguardEnabledV6":true}}' ``` > **SECONDARY:** The above command can be used to change other WireGuard attributes. For a list of other WireGuard parameters and configuration evaluation, see the [Felix configuration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig#felix-configuration-definition). We recommend that you review and modify the MTU used by Calico Cloud networking when WireGuard is enabled to increase network performance. Follow the instructions in the [Configure MTU to maximize network performance](https://docs.tigera.io/calico-cloud/networking/configuring/mtu) guide to set the MTU to a value appropriate for your network. ### Verify encryption is enabled[​](#verify-encryption-is-enabled) To verify that the nodes are configured for WireGuard encryption, check the node status set by Felix using `kubectl`. For example: ```bash kubectl get node -o yaml ... kind: Node metadata: annotations: projectcalico.org/WireguardPublicKey: jlkVyQYooZYzI2wFfNhSZez5eWh44yfq1wKVjLvSXgY= ... ``` ### Enable WireGuard statistics[​](#enable-wireguard-statistics) Since v3.11.1, WireGuard statistics are now automatically enabled with the enable wireguard setting(s) mentioned above. ### View WireGuard statistics[​](#view-wireguard-statistics) To view WireGuard statistics in the web console, you must enable them. From the left navbar, click **Dashboard**, and the Layout Settings icon. ![Wireguard Dashboard Toggle](https://docs.tigera.io/assets/images/stats-toggle-902e5bebc0151b882aeadd3cbb3f0dbc.png) ### Disable WireGuard for an individual node[​](#disable-wireguard-for-an-individual-node) To disable WireGuard on a specific node with WireGuard installed, modify the node-specific Felix configuration. e.g., to turn off encryption for pod traffic on node `my-node`, use the following command. This command disables WireGuard for both IPv4 and IPv6, modify it accordingly if disabling only either IP version: ```bash cat < **WARNING:** Calico Cloud assumes that you have exempted in-cluster destinations (including the Kubernetes API server) from proxying via the `NO_PROXY` variable. ## Configuration via Helm[​](#configuration-via-helm) If you wish to manually configure Calico Cloud via Helm to set proxy environment variables on Calico Cloud containers (rather than using a mutating admission webhook), you will first need to familiarize yourself with how to [customize the Calico Cloud Helm chart using values.yaml](https://docs.tigera.io/calico-cloud/get-started/install-automated#prepare-your-valuesyaml-with-customizations). Following the instructions linked above, retrieve your `.yaml`. Edit the file to set the `installer.components.installation.proxy` path, which configures the Installation CR with your proxy. This instructs the Tigera Operator to program your proxy on relevant containers that it deploys. e.g.: ```yaml installer: components: installation: proxy: httpProxy: httpsProxy: noProxy: ``` In your `.yaml` file, you'll also need to set the `installer.components.ccInstaller.operatorDeployment.spec.template.spec.containers` path, which programs your proxy on the calico-cloud-controller-manager pod. ```yaml installer: components: ccInstaller: operatorDeployment: spec: template: spec: containers: - name: calico-cloud env: - name: HTTP_PROXY value: - name: HTTPS_PROXY value: - name: NO_PROXY value: ``` Once you've completed your edits to your `.yaml`, you can [install](https://docs.tigera.io/calico-cloud/get-started/install-automated#install-calico-cloud-as-part-of-an-automated-workflow-1) Calico Cloud using your customized values. > **WARNING:** Ensure that the other required values for the Helm chart are set in the values.yaml file, as indicated by the generated helm install command. ## Configuration via Manifest[​](#configuration-via-manifest) If you wish to manually configure Calico Cloud via manifests to set proxy environment variables on Calico Cloud containers (rather than using a mutating admission webhook), you’ll need to edit the manifests generated at install and reinstall time. First, you’ll need to edit the cc-operator deploy.yaml so that the `calico-cloud/calico-cloud-controller-manager` Deployment resource has your required environment variables (`HTTP_PROXY`, `HTTPS_PROXY`, `NO_PROXY`) set in its `env` section. Second, you’ll need to edit the managed-cluster deploy.yaml so that the `calico-cloud/default` Installer resource has the Installation CR proxy configuration set. This is set at the `.spec.components.installation.proxy` path, e.g. ```text apiVersion: operator.calicocloud.io/v1 kind: Installer metadata: name: default namespace: calico-cloud spec: clusterName: "proxy-cluster" resourceVersion: "20241209202443" components: installation: proxy: httpProxy: httpsProxy: noProxy: ``` ## Validating Configuration[​](#validating-configuration) Once your install or reinstall has completed, you can: - Check the Installation CR to validate that the `.spec.proxy` path has been set as expected - Check the calico-cloud-controller-manager Deployment to validate that the environment variables have been set on the container as expected > **WARNING:** You can manually edit the two places above to configure Calico Cloud to use a proxy, but keep in mind that the reinstall process (via Helm or manifest) may overwrite your changes if you did not follow the steps in previous sections. --- ## Networking ### Networking The Calico Cloud network plugins provide a range of networking options to fit your implementation and maximize performance. ## Getting started[​](#getting-started) ##### [Networking overview](https://docs.tigera.io/calico-cloud/networking/training/about-networking) [Learn about networking layers, packets, IP addressing, and routing.](https://docs.tigera.io/calico-cloud/networking/training/about-networking) ##### [Kubernetes network model](https://docs.tigera.io/calico-cloud/networking/training/about-kubernetes-networking) [Learn network behaviors of the Kubernetes network model.](https://docs.tigera.io/calico-cloud/networking/training/about-kubernetes-networking) ## Configuring networking[​](#configuring-networking) ##### [Configure BGP peering](https://docs.tigera.io/calico-cloud/networking/configuring/bgp) [Configure BGP peering with full mesh, node-specific peering, ToR, and/or Calico route reflectors.](https://docs.tigera.io/calico-cloud/networking/configuring/bgp) ##### [Deploy a dual ToR cluster](https://docs.tigera.io/calico-cloud/networking/configuring/dual-tor) [Configure a dual plane cluster for redundant connectivity between workloads.](https://docs.tigera.io/calico-cloud/networking/configuring/dual-tor) ##### [Configure multiple Calico Cloud networks on a pod](https://docs.tigera.io/calico-cloud/networking/configuring/multiple-networks) [Configure a cluster with multiple Calico Cloud networks on each pod, and enforce security using Calico Cloud tiered network policy.](https://docs.tigera.io/calico-cloud/networking/configuring/multiple-networks) ##### [Overlay networking](https://docs.tigera.io/calico-cloud/networking/configuring/vxlan-ipip) [Configure Calico to use IP in IP or VXLAN overlay networking so the underlying network doesn’t need to understand pod addresses.](https://docs.tigera.io/calico-cloud/networking/configuring/vxlan-ipip) ##### [Advertise Kubernetes service IP addresses](https://docs.tigera.io/calico-cloud/networking/configuring/advertise-service-ips) [Configure Calico to advertise Kubernetes service cluster IPs and external IPs outside the cluster using BGP.](https://docs.tigera.io/calico-cloud/networking/configuring/advertise-service-ips) ##### [Configure MTU to maximize network performance](https://docs.tigera.io/calico-cloud/networking/configuring/mtu) [Optimize network performance for workloads by configuring the MTU in Calico to best suit your underlying network.](https://docs.tigera.io/calico-cloud/networking/configuring/mtu) ##### [Custom BGP Configuration](https://docs.tigera.io/calico-cloud/networking/configuring/custom-bgp-config) [Apply a custom BGP configuration](https://docs.tigera.io/calico-cloud/networking/configuring/custom-bgp-config) ##### [Configure outgoing NAT](https://docs.tigera.io/calico-cloud/networking/configuring/workloads-outside-cluster) [Configure Calico Cloud networking to perform outbound NAT for connections from pods to outside of the cluster.](https://docs.tigera.io/calico-cloud/networking/configuring/workloads-outside-cluster) ##### [Use a specific MAC address for a pod](https://docs.tigera.io/calico-cloud/networking/configuring/pod-mac-address) [Specify the MAC address for a pod instead of allowing the operating system to assign one](https://docs.tigera.io/calico-cloud/networking/configuring/pod-mac-address) ##### [Use NodeLocal DNSCache in your cluster](https://docs.tigera.io/calico-cloud/networking/configuring/node-local-dns-cache) [Install NodeLocal DNSCache](https://docs.tigera.io/calico-cloud/networking/configuring/node-local-dns-cache) ## IP address management[​](#ip-address-management) ##### [Configure BGP peering](https://docs.tigera.io/calico-cloud/networking/configuring/bgp) [Configure BGP peering with full mesh, node-specific peering, ToR, and/or Calico route reflectors.](https://docs.tigera.io/calico-cloud/networking/configuring/bgp) ##### [Deploy a dual ToR cluster](https://docs.tigera.io/calico-cloud/networking/configuring/dual-tor) [Configure a dual plane cluster for redundant connectivity between workloads.](https://docs.tigera.io/calico-cloud/networking/configuring/dual-tor) ##### [Configure multiple Calico Cloud networks on a pod](https://docs.tigera.io/calico-cloud/networking/configuring/multiple-networks) [Configure a cluster with multiple Calico Cloud networks on each pod, and enforce security using Calico Cloud tiered network policy.](https://docs.tigera.io/calico-cloud/networking/configuring/multiple-networks) ##### [Overlay networking](https://docs.tigera.io/calico-cloud/networking/configuring/vxlan-ipip) [Configure Calico to use IP in IP or VXLAN overlay networking so the underlying network doesn’t need to understand pod addresses.](https://docs.tigera.io/calico-cloud/networking/configuring/vxlan-ipip) ##### [Advertise Kubernetes service IP addresses](https://docs.tigera.io/calico-cloud/networking/configuring/advertise-service-ips) [Configure Calico to advertise Kubernetes service cluster IPs and external IPs outside the cluster using BGP.](https://docs.tigera.io/calico-cloud/networking/configuring/advertise-service-ips) ##### [Configure MTU to maximize network performance](https://docs.tigera.io/calico-cloud/networking/configuring/mtu) [Optimize network performance for workloads by configuring the MTU in Calico to best suit your underlying network.](https://docs.tigera.io/calico-cloud/networking/configuring/mtu) ##### [Custom BGP Configuration](https://docs.tigera.io/calico-cloud/networking/configuring/custom-bgp-config) [Apply a custom BGP configuration](https://docs.tigera.io/calico-cloud/networking/configuring/custom-bgp-config) ##### [Configure outgoing NAT](https://docs.tigera.io/calico-cloud/networking/configuring/workloads-outside-cluster) [Configure Calico Cloud networking to perform outbound NAT for connections from pods to outside of the cluster.](https://docs.tigera.io/calico-cloud/networking/configuring/workloads-outside-cluster) ##### [Use a specific MAC address for a pod](https://docs.tigera.io/calico-cloud/networking/configuring/pod-mac-address) [Specify the MAC address for a pod instead of allowing the operating system to assign one](https://docs.tigera.io/calico-cloud/networking/configuring/pod-mac-address) ##### [Use NodeLocal DNSCache in your cluster](https://docs.tigera.io/calico-cloud/networking/configuring/node-local-dns-cache) [Install NodeLocal DNSCache](https://docs.tigera.io/calico-cloud/networking/configuring/node-local-dns-cache) ##### [LoadBalancer IP address management](https://docs.tigera.io/calico-cloud/networking/ipam/service-loadbalancer) [LoadBalancer IP address management](https://docs.tigera.io/calico-cloud/networking/ipam/service-loadbalancer) ## Egress gateways[​](#egress-gateways) ##### [Configure egress gateways, on-premises](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-on-prem) [Configure specific application traffic to exit the cluster through an egress gateway.](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-on-prem) ##### [Configure egress gateways, Azure](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-azure) [Configure specific application traffic to exit the cluster through an egress gateway with a native Azure IP address.](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-azure) ##### [Configure egress gateways, AWS](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-aws) [Configure specific application traffic to exit the cluster through an egress gateway with a native AWS IP address.](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-aws) ##### [Optimize egress networking for workloads with long-lived TCP connections](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-maintenance) [React to egress gateway maintenance windows and minimize the impact of egress gateway downtime on sensitive workloads](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-maintenance) ##### [Configure egress traffic to multiple external networks](https://docs.tigera.io/calico-cloud/networking/egress/external-network) [Allows workloads from different namespaces of a Kubernetes cluster to egress onto different external networks that (may) have overlapping IPs with each other.](https://docs.tigera.io/calico-cloud/networking/egress/external-network) ##### [Troubleshoot egress gateways](https://docs.tigera.io/calico-cloud/networking/egress/troubleshoot) [Use checklist to troubleshoot common problems.](https://docs.tigera.io/calico-cloud/networking/egress/troubleshoot) ### Calico Ingress Gateway Calico Ingress Gateway provides a streamlined solution for managing cluster ingress traffic by using the Kubernetes Gateway API standard. ## About Calico Ingress Gateway[​](#about-calico-ingress-gateway) Calico Ingress Gateway is a hardened distribution of the open-source Envoy Gateway project. Tigera rebuilds the Envoy Gateway code using hardened base images for robust security and stable performance. Calico Ingress Gateway provides seamless, deep integration with the Calico environment. This approach ensures that ingress security and networking configurations are unified within the Calico framework. ## About Gateway API[​](#about-gateway-api) The Kubernetes Gateway API is the modern, standardized interface for governing access to services within a cluster, serving as the successor to the original Ingress resource. Key features and benefits of the Gateway API include: - Role-oriented design: It clearly separates network infrastructure concerns (managed by network teams via Gateway resources) from application-specific routing (managed by developers via routing resources). - Advanced configuration: The API natively supports advanced traffic control patterns, such as weighted traffic splitting for canary deployments, header manipulation, and detailed request matching, functionality that often required vendor-specific annotations in the past. - Portability: As a standard Kubernetes API, configurations are not tied to a single vendor implementation, providing portability and preventing vendor lock-in. ## About Envoy Gateway[​](#about-envoy-gateway) Gateway API defines the specification, and Envoy Gateway is the primary implementation framework. Envoy Gateway acts as the control plane that translates Gateway API resources into actionable configuration for its data plane. The data plane utilizes Envoy Proxy, a high-performance, widely adopted proxy used across the cloud-native ecosystem for edge and service mesh functionality. By leveraging Envoy, the Calico Ingress Gateway provides proven reliability, high throughput, and granular observability. ## Deployment and management with Calico Ingress Gateway[​](#deployment-and-management-with-calico-ingress-gateway) While the underlying architecture involves multiple layers (Gateway API specification, Envoy Gateway control plane, Envoy Proxy data plane), the Calico Ingress Gateway simplifies deployment and management significantly. Calico Ingress Gateway is managed entirely by the Tigera Operator. This integration automates the lifecycle of the components to reduce manual configuration overhead. CVEs are addressed as part of the regular Calico Cloud patch release cadence. Administrators provision the gateway environment simply by defining a standard Gateway resource. ### Calico Enterprise and Calico Cloud integration[​](#calico-enterprise-and-calico-cloud-integration) For commercial deployments, Calico Ingress Gateway extends functionality by integrating directly with the Web Application Firewall (WAF). This feature allows operators to secure their ingress points against the OWASP Top 10 and other common vulnerabilities through simple configuration within the Calico management plane. ## Supported Gateway API resources[​](#supported-gateway-api-resources) Calico Ingress Gateway supports the following Gateway API resources. | Resource | Versions | | ---------------- | ----------------- | | BackendLBPolicy | v1alpha2 | | BackendTLSPolicy | v1alpha3 | | GatewayClass | v1, v1beta1 | | Gateway | v1, v1beta1 | | GRPCRoute | v1, v1alpha2 | | HTTPRoute | v1, v1beta1 | | ReferenceGrant | v1beta1, v1alpha2 | | TCPRoute | v1alpha2 | | TLSRoute | v1alpha2 | | UDPRoute | v1alpha2 | > **SECONDARY:** OpenShift 4.19 introduced [restrictions](https://docs.redhat.com/en/documentation/openshift_container_platform/4.19/html-single/ingress_and_load_balancing/index#nw-ingress-gateway-api-overview_ingress-gateway-api) on which Gateway API CRDs can be installed. > > - The available Gateway API resources in OpenShift 4.19 are: `GatewayClass`, `Gateway`, `GRPCRoute`, `HTTPRoute` and `ReferenceGrant`. > > - Other Gateway API resources, namely `BackendLBPolicy`, `BackendTLSPolicy`, `TCPRoute`, `TLSRoute` and `UDPRoute`, are not available in OpenShift 4.19. ## Next steps[​](#next-steps) - Learn more about Calico Ingress Gateway with the [canary deployment tutorial](https://docs.tigera.io/calico-cloud/networking/ingress-gateway/tutorial-ingress-gateway-canary). ## Additional resources[​](#additional-resources) - [Kubernetes Gateway API documentation](https://gateway-api.sigs.k8s.io/) - [Envoy Gateway documentation](https://gateway.envoyproxy.io/docs/) ### Create an ingress gateway Create an ingress gateway to manage ingress traffic with the Kubernetes Gateway API. ## Creating an ingress gateway with Calico Ingress Gateway[​](#creating-an-ingress-gateway-with-calico-ingress-gateway) Calico Ingress Gateway is based on Envoy Gateway, an open-source implementation of the Gateway API. Calico provides a simple, convenient way to deploy and manage this implementation. Creating an ingress gateway with Calico Ingress Gateway involves only a few steps. You need to do the following: - Enable Calico Ingress Gateway by creating a `GatewayAPI` resource. - Create a `Gateway` resource that references `tigera-gateway-class`, the name of the `GatewayClass` resource created by the Tigera Operator. - When you create your gateway routing resources, you refer to this `Gateway` resource. ## Limitations[​](#limitations) - Calico Ingress Gateway is not supported on Windows nodes. ## Prerequisites[​](#prerequisites) - You have an understanding of the [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io/) and know how you want to configure it for your system. - Calico Cloud was installed on your cluster using the Tigera Operator. - Your cluster is provisioned to support services of type `LoadBalancer`. ## Create an ingress gateway[​](#create-an-ingress-gateway-1) 1. To enable Gateway API support, create a `GatewayAPI` resource with the name `tigera-secure`: ```bash kubectl apply -f - < **SUCCESS:** `tigera-gateway-class` is the name of the single default `GatewayClass` resource that is created when you enable Gateway API support. You can define [additional gateway classes](https://docs.tigera.io/calico-cloud/networking/ingress-gateway/customize-ingress-gateway#configure-multiple-gateway-classes), along with other customizations, in the `GatewayAPI` resource. 2. Create a `Gateway` resource that is linked to `tigera-gateway-class`. Example snippet of a Gateway resource with default gatewayClassName ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: spec: gatewayClassName: tigera-gateway-class ... ``` Replace `` with a name for your gateway. You will refer to this gateway name for all services you want to use the gateway. 3. Create a gateway routing resource that refers to your `Gateway` resource as `.spec.parentRefs`: Example snippet of an HTTPRoute resource ```yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: example-httproute spec: parentRefs: - name: ... ``` Replace `` with the name of your `Gateway` resource. ## Additional resources[​](#additional-resources) - [Kubernetes Gateway API documentation](https://gateway-api.sigs.k8s.io/) - [Envoy Gateway documentation](https://gateway.envoyproxy.io/docs/) - [Tutorial: Launch a canary deployment](https://docs.tigera.io/calico-cloud/networking/ingress-gateway/tutorial-ingress-gateway-canary) ### Customizing your ingress gateway You can specify core configuration elements of your ingress gateway by specifying a configuration in the `GatewayAPI` resource. Alternatively, for the complete set of customization options available to Envoy Gateway, you can provide your own Envoy Gateway configuration. ## Customizing an ingress gateway with the `GatewayAPI` resource[​](#customizing-an-ingress-gateway-with-the-gatewayapi-resource) Many customizations are available for the `GatewayAPI` resource. This resource has fields that allow some aspects of Gateway deployments to be customized. For example: - `spec.gatewayDeployment.spec.template.metadata` allows arbitrary labels or annotations to be added to the pod that is created to implement each configured Gateway. - `spec.gatewayDeployment.spec.template.spec.nodeSelector` allows control over where gateway implementation pods are scheduled. - `spec.gatewayDeployment.spec.template.spec.containers` allows control over the memory and CPU that the gateway implementation pods can use. - `spec.gatewayControllerDeployment.spec.template.spec.nodeSelector` allows control over where the gateway controller is scheduled. - `spec.gatewayDeployment.service.metadata.annotations` allows control over annotations to place on the service that is provisioned for each Gateway; these can be used, for example, to configure the cloud-specific type and properties of the associated external load balancer. - `spec.gatewayDeployment.service.spec.*loadbalancer*` allows control over the corresponding `*loadbalancer*` fields in the service that is provisioned for each gateway; in some clouds these can also be used to configure the type and properties of the associated external load balancer. - `spec.gatewayClasses` allows the provisioning of multiple `GatewayClass` resources, each with its own set of customizations, instead of the single `tigera-gateway-class` gateway class that the Tigera Operator provisions by default. For full details, see [the `GatewayAPI` reference documentation](https://docs.tigera.io/calico-cloud/reference/installation/api#gatewayapi). To make use of these customization fields, use `kubectl edit gatewayapi tigera-secure` to edit the YAML for the `GatewayAPI` resource, and add or modify the customization fields that you require. ### Customization examples[​](#customization-examples) This section contains examples of common customizations. #### Handle high traffic with multiple gateway pods and adequate memory per-pod[​](#handle-high-traffic-with-multiple-gateway-pods-and-adequate-memory-per-pod) ```bash kubectl apply -f - < .yaml ``` Replace `` with a name for your custom configuration. Example of a default Envoy configuration ```yaml apiVersion: v1 data: envoy-gateway.yaml: | apiVersion: gateway.envoyproxy.io/v1alpha1 extensionApis: enableBackend: true enableEnvoyPatchPolicy: true gateway: controllerName: gateway.envoyproxy.io/gatewayclass-controller kind: EnvoyGateway logging: level: default: info provider: kubernetes: rateLimitDeployment: container: image: quay.io/calico/envoy-ratelimit:v3.31.1 patch: type: StrategicMerge value: spec: template: spec: containers: - imagePullPolicy: IfNotPresent name: envoy-ratelimit pod: {} shutdownManager: image: quay.io/calico/envoy-gateway:v3.31.1 type: Kubernetes kind: ConfigMap metadata: creationTimestamp: "2025-11-18T12:34:04Z" labels: app.kubernetes.io/instance: tigera-gateway-api app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: gateway-helm app.kubernetes.io/version: v1.5.0 helm.sh/chart: gateway-helm-v1.5.0 name: envoy-gateway-config namespace: tigera-gateway ownerReferences: - apiVersion: operator.tigera.io/v1 blockOwnerDeletion: true controller: true kind: GatewayAPI name: default uid: ace36d13-3afc-4c52-b894-dc2b322bc36e resourceVersion: "1978" uid: 8135e15d-cc91-4e02-ab97-2910cff56008 ``` 2. Remove the following cluster-generated metadata fields: - `creationTimestamp` - `ownerReferences` - `resourceVersion` - `uid` 3. Change `metadata.name` from the default `envoy-gateway-config` to something else. 4. Add your customizations to the `data.envoy-gateway.yaml` section. Example of a customized Envoy configuration file with Jaeger integration ```yaml apiVersion: v1 data: envoy-gateway.yaml: | apiVersion: gateway.envoyproxy.io/v1alpha1 extensionApis: enableBackend: true enableEnvoyPatchPolicy: true gateway: controllerName: gateway.envoyproxy.io/gatewayclass-controller kind: EnvoyGateway logging: level: default: info provider: kubernetes: rateLimitDeployment: container: image: quay.io/calico/envoy-ratelimit:v3.31.1 patch: type: StrategicMerge value: spec: template: spec: containers: - imagePullPolicy: IfNotPresent name: envoy-ratelimit pod: {} shutdownManager: image: quay.io/calico/envoy-gateway:v3.31.1 type: Kubernetes tracing: enabled: true provider: type: Jaeger jaeger: host: jaeger-collector.observability.svc.cluster.local port: 14268 kind: ConfigMap metadata: labels: app.kubernetes.io/instance: tigera-gateway-api app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: gateway-helm app.kubernetes.io/version: v1.5.0 helm.sh/chart: gateway-helm-v1.5.0 name: custom-envoy-gateway-config namespace: tigera-gateway ``` 5. Apply the customized Envoy configuration: ```bash kubectl apply -f custom-envoy-config.yaml ``` 6. Patch the `GatewayAPI` resource to include the path of the custom Envoy configuration: ```bash kubectl patch gatewayapi tigera-secure --type='json' -p='[{"op": "replace", "path": "/spec/envoyGatewayConfigRef", "value": {"name": "custom-envoy-config", "namespace": "tigera-gateway"}}]' ``` ## Additional resources[​](#additional-resources) - `GatewayAPI` spec in the [Installation API documentation](https://docs.tigera.io/calico-cloud/reference/installation/api#gatewayapi) - [Envoy Gateway documentation](https://gateway.envoyproxy.io/docs/) ### Tutorial: Launch a canary deployment with Calico Ingress Gateway This tutorial shows you how to create a canary deployment by using Calico Ingress Gateway. ## Overview[​](#overview) A **canary deployment** is a progressive release strategy where a new version of an application is gradually rolled out to a small subset of users before being fully deployed. This approach allows teams to detect issues early, validate new features, and roll back quickly if needed, minimizing disruption to the majority of users. In Calico Cloud, canary deployments are implemented with a **traffic splitting** method that is defined in a routing resource associated with an ingress gateway. By defining routing rules, you direct a small proportion of your incoming traffic to the canary version of your workload for testing. The majority of the traffic continues to the stable version. As you become more confident in the canary version, you increase the proportion of traffic going to that version incrementally until it becomes the new stable version. ## Use cases[​](#use-cases) - **Feature Rollouts:** Safely release a new feature to a small subset of users to validate functionality and gather feedback before full deployment. - **Performance Testing:** Evaluate how a new version handles real-world traffic and resource usage without affecting all users. - **Bug Detection:** Catch critical bugs or unexpected behavior in production with minimal user impact. - **A/B Testing:** Test different versions of a service to compare performance, user engagement, or conversion rates before committing to one. - **Compliance and Risk Management:** Gradually roll out updates in highly regulated or mission-critical environments to reduce the chance of widespread failures. Canary deployments are ideal when you want controlled, low-risk releases, allowing teams to monitor metrics, validate assumptions, and ensure stability before a full-scale rollout. ## Before you begin[​](#before-you-begin) You'll need to install a few tools to complete this tutorial: - `kind`. This is what you'll use to create a cluster on your workstation. For installation instructions, see the [`kind` documentation](https://kind.sigs.k8s.io/docs/user/quick-start/). - Docker Engine or Docker Desktop. This is required to run containers for the `kind` utility. For installation instructions, see the [Docker documentation](https://docs.docker.com/desktop/). - `kubectl`. This is the tool you'll use to interact with your cluster. For installation instructions, see the [Kubernetes documentation](https://kubernetes.io/docs/tasks/tools/#kubectl) ## Step 1: Set up your environment[​](#step-1-set-up-your-environment) For this tutorial, you need access to a cluster with Calico Open Source 3.30 or later installed. We'll use `kind` to create this environment, but you can use any other supported Kubernetes distribution with a compatible version of Calico Cloud. If you've already got a suitable environment, start the tutorial at [Step 2: Create the ingress gateway](#step-2-create-an-ingress-gateway). 1. Create a `kind` cluster with one control-plane node and two worker nodes. ```bash cat > config.yaml < 5m23s v1.33.1 calico-cluster-worker2 NotReady 5m22s v1.33.1 ``` The nodes remain in a `NotReady` status until you configure networking in the next step. 3. Install Calico Open Source by adding custom resource definitions, the Tigera Operator, and the custom resources. ```text kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.31.1/manifests/operator-crds.yaml kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.31.1/manifests/tigera-operator.yaml kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.31.1/manifests/custom-resources.yaml ``` 4. Monitor the deployment by running the following command: ```bash watch kubectl get tigerastatus ``` After a few minutes, all the Calico components display `True` in the `AVAILABLE` column. Expected output ```bash NAME AVAILABLE PROGRESSING DEGRADED SINCE apiserver True False False 4m9s calico True False False 3m29s goldmane True False False 3m39s ippools True False False 6m4s whisker True False False 3m19s ``` ## Step 2. Create an ingress gateway[​](#step-2-create-an-ingress-gateway) Enable the Gateway API by creating a `GatewayAPI` resource. When you create this resource, the Tigera Operator pulls Envoy Gateway images and sets things up. Once that is set up, you can create supported Gateway API resources, such as the `Gateway` and routing resources. 1. To enable Gateway API support, create a `GatewayAPI` resource with name `default`: ```bash kubectl apply -f - <

App Version 1

--- apiVersion: apps/v1 kind: Deployment metadata: name: app-v1 namespace: ingress-gateway-demo spec: replicas: 1 selector: matchLabels: app: ingress-gateway-demo version: v1 template: metadata: labels: app: ingress-gateway-demo version: v1 spec: containers: - name: busybox image: busybox:latest command: ["sh", "-c", "httpd -f -p 80 -h /var/www/html"] ports: - containerPort: 80 volumeMounts: - name: html mountPath: /var/www/html volumes: - name: html configMap: name: app-v1-html --- apiVersion: v1 kind: Service metadata: name: app-v1 namespace: ingress-gateway-demo spec: selector: app: ingress-gateway-demo version: v1 ports: - port: 80 targetPort: 80 EOF ``` Expected output ```bash configmap/app-v1-html created deployment.apps/app-v1 created service/app-v1 created ``` 3. Now deploy the same resources for the canary build, ingress-gateway-demo v2: ```bash cat << EOF | kubectl apply -f - apiVersion: v1 kind: ConfigMap metadata: name: app-v2-html namespace: ingress-gateway-demo data: index.html: |

App Version 2

--- apiVersion: apps/v1 kind: Deployment metadata: name: app-v2 namespace: ingress-gateway-demo spec: replicas: 1 selector: matchLabels: app: ingress-gateway-demo version: v2 template: metadata: labels: app: ingress-gateway-demo version: v2 spec: containers: - name: busybox image: busybox:latest command: ["sh", "-c", "httpd -f -p 80 -h /var/www/html"] ports: - containerPort: 80 volumeMounts: - name: html mountPath: /var/www/html volumes: - name: html configMap: name: app-v2-html --- apiVersion: v1 kind: Service metadata: name: app-v2 namespace: ingress-gateway-demo spec: selector: app: ingress-gateway-demo version: v2 ports: - port: 80 targetPort: 80 EOF ``` Expected output ```bash configmap/app-v2-html created deployment.apps/app-v2 created service/app-v2 created ``` ## Step 4: Define an `HTTPRoute` resource to split traffic between app-v1 and app-v2[​](#step-4-define-an-httproute-resource-to-split-traffic-between-app-v1-and-app-v2) The `HTTPRoute` below routes 80% of requests to app-v1 and 20% to app-v2 ```bash cat << EOF | kubectl apply -f - apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: traffic-splitting spec: parentRefs: - name: canary-deployment-gateway namespace: default rules: - matches: - path: type: PathPrefix value: / backendRefs: - name: app-v1 namespace: ingress-gateway-demo port: 80 weight: 80 - name: app-v2 namespace: ingress-gateway-demo port: 80 weight: 20 EOF ``` ## Step 5. Allow ingress traffic from the gateway to reach services in the `ingress-gateway-demo` namespace[​](#step-5-allow-ingress-traffic-from-the-gateway-to-reach-services-in-the-ingress-gateway-demo-namespace) You can permit traffic to be routed from the gateway to your app by adding definitions to a `ReferenceGrant` resource. ```bash kubectl apply -f - <"; done | \ awk ' /App Version 1/ {v1++} /App Version 2/ {v2++} END { total=v1+v2 printf("Version 1: %d (%.1f%%)\nVersion 2: %d (%.1f%%)\n", v1, v1/total*100, v2, v2/total*100) }' ``` Sample output ```bash Version 1: 161 (80.5%) Version 2: 39 (19.5%) ``` ## Step 7. Clean up your environment[​](#step-7-clean-up-your-environment) - To clean up your tutorial environment, run the following command: ```bash kind delete cluster --name calico-cluster ``` ### Networking basics ## [📄️ Networking overview](https://docs.tigera.io/calico-cloud/networking/training/about-networking) [Learn about networking layers, packets, IP addressing, and routing.](https://docs.tigera.io/calico-cloud/networking/training/about-networking) ## [📄️ Kubernetes network model](https://docs.tigera.io/calico-cloud/networking/training/about-kubernetes-networking) [Learn network behaviors of the Kubernetes network model.](https://docs.tigera.io/calico-cloud/networking/training/about-kubernetes-networking) ### Networking overview > **SECONDARY:** This guide provides educational material that is not specific to Calico Cloud. You can get up and running with Calico Cloud by following any of the Calico Cloud install guides without needing to be a networking expert. Calico Cloud hides the complexities for you. However, if you would like to learn more about networking so you can better understand what is happening under the covers, this guide provides a short introduction to some of the key fundamental networking concepts for anyone who is not already familiar with them. In this guide you will learn: - The terms used to described different layers of the network. - The anatomy of a network packet. - What MTU is and why it makes a difference. - How IP addressing, subnets, and IP routing works. - What an overlay network is. - What DNS and NAT are. ## Network layers[​](#network-layers) The process of sending and receiving data over a network is commonly categorized into 7 layers (referred to as the [OSI model](https://en.wikipedia.org/wiki/OSI_model)). The layers are typically abbreviated as L1 - L7. You can think of data as passing through each of these layers in turn as it is sent or received from an application, with each layer being responsible for a particular part of the processing required to send or receive the data over the network. ![OSI network layers diagram](https://docs.tigera.io/assets/images/osi-network-layers-783163dbb30a9c3f6ffd9014c1423276.svg) In a modern enterprise or public cloud network, the layers commonly map as follows: - L5-7: all the protocols most application developers are familiar with. e.g. HTTP, FTP, SSH, SSL, DNS. - L4: TCP or UDP, including source and destination ports. - L3: IP packets and IP routing. - L2: Ethernet packets and Ethernet switching. ## Anatomy of a network packet[​](#anatomy-of-a-network-packet) When sending data over the network, each layer in the network stack adds its own header containing the control/metadata the layer needs to process the packet as it traverses the network, passing the resulting packet on to the next layer of the stack. In this way the complete packet is produced, which includes all the control/metadata required by every layer of the stack, without any layer understanding the data or needing to process the control/metadata of adjacent network layers. ![Anatomy of a network packet](https://docs.tigera.io/assets/images/anatomy-of-a-packet-e5feeb4df211714f48278f8e7ba4645b.svg) ## IP addressing, subnets and IP routing[​](#ip-addressing-subnets-and-ip-routing) The L3 network layer introduces IP addresses and typically marks the boundary between the part of networking that application developers care about, and the part of networking that network engineers care about. In particular application developers typically regard IP addresses as the source and destination of the network traffic, but have much less of a need to understand L3 routing or anything lower in the network stack, which is more the domain of network engineers. There are two variants of IP addresses: IPv4 and IPv6. - IPv4 addresses are 32 bits long and the most commonly used. They are typically represented as 4 bytes in decimal (each 0-255) separated by dots. e.g. `192.168.27.64`. There are several ranges of IP addresses that are reserved as "private", that can only be used within local private networks, are not routable across the internet. These can be reused by enterprises as often as they want to. In contrast "public" IP addresses are globally unique across the whole of the internet. As the number of network devices and networks connected to the internet has grown, public IPv4 addresses are now in short supply. - IPv6 addresses are 128 bits long and designed to overcome the shortage of IPv4 address space. They are typically represented by 8 groups of 4 digit hexadecimal numbers. e.g. `1203:8fe0:fe80:b897:8990:8a7c:99bf:323d`. Due to the 128 bit length, there's no shortage of IPv6 addresses. However, many enterprises have been slow to adopt IPv6, so for now at least, IPv4 remains the default for many enterprise and data center networks. Groups of IP addresses are typically represented using [CIDR notation](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing) that consists of an IP address and number of significant bits on the IP address separated by a `/`. For example, `192.168.27.0/24` represents the group of 256 IP addresses from `192.168.27.0` to `192.168.27.255`. A group of IP addresses within a single L2 network is referred to as a subnet. Within a subnet, packets can be sent between any pair of devices as a single network hop, based solely on the L2 header (and footer). To send packets beyond a single subnet requires L3 routing, with each L3 network device (router) being responsible for making decisions on the path to send the packet based on L3 routing rules. Each network device acting as a router has routes that determine where a packet for a particular CIDR should be sent next. So for example, in a Linux system, a route of `10.48.0.128/26 via 10.0.0.12 dev eth0` indicates that packets with destination IP address in `10.48.0.128/26` should be routed to a next network hop of `10.0.0.12` over the `eth0` interface. Routes can be configured statically by an administrator, or programmed dynamically using routing protocols. When using routing protocols each network device typically needs to be configured to tell it which other network devices it should be exchanging routes with. The routing protocol then handles programming the right routes across the whole of the network as devices are added or removed, or network links come in or out of service. One common routing protocol used in large enterprise and data center networks is [BGP](https://en.wikipedia.org/wiki/Border_Gateway_Protocol). BGP is one of the main protocols that powers the internet, so scales incredibly well, and is very widely supported by modern routers. ## Overlay networks[​](#overlay-networks) An overlay network allows network devices to communicate across an underlying network (referred to as the underlay) without the underlay network having any knowledge of the devices connected to the overlay network. From the point of view of the devices connected to the overlay network, it looks just like a normal network. There are many different kinds of overlay networks that use different protocols to make this happen, but in general they share the same common characteristic of taking a network packet, referred to as the inner packet, and encapsulating it inside an outer network packet. In this way the underlay sees the outer packets without needing to understand how to handle the inner packets. How the overlay knows where to send packets varies by overlay type and the protocols they use. Similarly exactly how the packet is wrapped varies between different overlay types. In the case of VXLAN for example, the inner packet is wrapped and sent as UDP in the outer packet. ![Anatomy of an overlay network packet](https://docs.tigera.io/assets/images/anatomy-of-an-overlay-packet-83230b9fc48c864b8223cead52d36e86.svg) Overlay networks have the advantage of having minimal dependencies on the underlying network infrastructure, but have the downsides of: - having a small performance impact compared to non-overlay networking, which you might want to avoid if running network intensive workloads - workloads on the overlay are not easily addressable from the rest of the network. so NAT gateways or load balancers are required to bridge between the overlay and the underlay network for any ingress to, or egress from, the overlay. Calico Cloud networking options are exceptionally flexible, so in general you can choose whether you prefer Calico Cloud to provide an overlay network, or non-overlay network. ## DNS[​](#dns) While the underlying network packet flow across the network is determined using IP addresses, users and applications typically want to use well known names to identify network destinations that remain consistent over time, even if the underlying IP addresses change. For example, to map `google.com` to `216.58.210.46`. This translation from name to IP address is handled by [DNS](https://en.wikipedia.org/wiki/Domain_Name_System). DNS runs on top of the base networking described so far. Each device connected to a network is typically configured with the IP addresses of one or more DNS servers. When an application wants to connect to a domain name, a DNS message is sent to the DNS server, which then responds with information about which IP address(es) the domain name maps to. The application can then initiate its connection to the chosen IP address. ## NAT[​](#nat) Network Address Translation ([NAT](https://en.wikipedia.org/wiki/Network_address_translation)) is the process of mapping an IP address in a packet to a different IP address as the packet passes through the device performing the NAT. Depending on the use case, NAT can apply to the source or destination IP address, or to both addresses. One common use case for NAT is to allow devices with private IP address to talk to devices with public IP address across the internet. For example, if a device with a private IP address attempts to connect to a public IP address, then the router at the border of the private network will typically use SNAT (Source Network Address Translation) to map the private source IP address of the packet to the router's own public IP address before forwarding it on to the internet. The router then maps response packets coming in the opposite direction back to the original private IP address, so packets flow end-to-end in both directions, with neither source or destination being aware the mapping is happening. The same technique is commonly used to allow devices connected to an overlay network to connect with devices outside of the overlay network. Another common use case for NAT is load balancing. In this case the load balancer performs DNAT (Destination Network Address Translation) to change the destination IP address of the incoming connection to the IP address of the chosen device it is load balancing to. The load balancer then reverses this NAT on response packets so neither source or destination device is aware the mapping is happening. ## MTU[​](#mtu) The Maximum Transmission Unit ([MTU](https://en.wikipedia.org/wiki/Maximum_transmission_unit)) of a network link is the maximum size of packet that can be sent across that network link. It is common for all links in a network to be configured with the same MTU to reduce the need to fragment packets as they traverse the network, which can significantly lower the performance of the network. In addition, TCP tries to learn path MTUs, and adjust packet sizes for each network path based on the smallest MTU of any of the links in the network path. When an application tries to send more data than can fit in a single packet, TCP will fragment the data into multiple TCP segments, so the MTU is not exceeded. Most networks have links with an MTU of 1,500 bytes, but some networks support MTUs of 9,000 bytes. In a Linux system, larger MTU sizes can result in lower CPU being used by the Linux networking stack when sending large amounts of data, because it has to process fewer packets for the same amount of data. Depending on the network interface hardware being used, some of this overhead may be offloaded to the network interface hardware, so the impact of small vs large MTU sizes varies from device to device. ### Kubernetes network model > **SECONDARY:** This guide provides educational material that is not specific to Calico Cloud. Kubernetes defines a network model that helps provide simplicity and consistency across a range of networking environments and network implementations. The Kubernetes network model provides the foundation for understanding how containers, pods, and services within Kubernetes communicate with each other. This guide explains the key concepts and how they fit together. In this guide you will learn: - The fundamental network behaviors the Kubernetes network model defines. - How Kubernetes works with a variety of different network implementations. - What Kubernetes Services are. - How DNS works within Kubernetes. - What "NAT outgoing" is and when you would want to use it. - What "dual stack" is. ## The Kubernetes network model[​](#the-kubernetes-network-model) The Kubernetes network model specifies: - Every pod gets its own IP address - Containers within a pod share the pod IP address and can communicate freely with each other - Pods can communicate with all other pods in the cluster using pod IP addresses (without [NAT](https://docs.tigera.io/calico-cloud/networking/training/about-networking)) - Isolation (restricting what each pod can communicate with) is defined using network policies As a result, pods can be treated much like VMs or hosts (they all have unique IP addresses), and the containers within pods very much like processes running within a VM or host (they run in the same network namespace and share an IP address). This model makes it easier for applications to be migrated from VMs and hosts to pods managed by Kubernetes. In addition, because isolation is defined using network policies rather than the structure of the network, the network remains simple to understand. This style of network is sometimes referred to as a "flat network". Note that, although very rarely needed, Kubernetes does also support the ability to map host ports through to pods, or to run pods directly within the host network namespace sharing the host's IP address. ## Kubernetes network implementations[​](#kubernetes-network-implementations) Kubernetes built in network support, kubenet, can provide some basic network connectivity. However, it is more common to use third party network implementations which plug into Kubernetes using the CNI (Container Network Interface) API. There are lots of different kinds of CNI plugins, but the two main ones are: - network plugins, which are responsible for connecting pod to the network - IPAM (IP Address Management) plugins, which are responsible for allocating pod IP addresses. Calico Cloud provides both network and IPAM plugins, but can also integrate and work seamlessly with some other CNI plugins, including AWS, Azure, and Google network CNI plugins, and the host local IPAM plugin. This flexibility allows you to choose the best networking options for your specific needs and deployment environment. ## Kubernetes Services[​](#kubernetes-services) Kubernetes [Services](https://kubernetes.io/docs/concepts/services-networking/service/) provide a way of abstracting access to a group of pods as a network service. The group of pods is usually defined using a [label selector](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels) . Within the cluster the network service is usually represented as virtual IP address, and kube-proxy load balances connections to the virtual IP across the group of pods backing the service. The virtual IP is discoverable through Kubernetes DNS. The DNS name and virtual IP address remain constant for the life time of the service, even though the pods backing the service may be created or destroyed, and the number of pods backing the service may change over time. Kubernetes Services can also define how a service is accessed from outside of the cluster, for example using - a node port, where the service can be accessed via a specific port on every node - or a load balancer, whether a network load balancer provides a virtual IP address that the service can be accessed via from outside the cluster. Note that when using Calico Cloud in on-prem deployments you can also [advertise service IP addresses](https://docs.tigera.io/calico-cloud/networking/configuring/advertise-service-ips) , allowing services to be conveniently accessed without going via a node port or load balancer. ## Kubernetes DNS[​](#kubernetes-dns) Each Kubernetes cluster provides a DNS service. Every pod and every service is discoverable through the Kubernetes DNS service. For example: - Service: `my-svc.my-namespace.svc.cluster-domain.example` - Pod: `pod-ip-address.my-namespace.pod.cluster-domain.example` - Pod created by a deployment exposed as a service: `pod-ip-address.deployment-name.my-namespace.svc.cluster-domain.example`. The DNS service is implemented as Kubernetes Service that maps to one or more DNS server pods (usually CoreDNS), that are scheduled just like any other pod. Pods in the cluster are configured to use the DNS service, with a DNS search list that includes the pod's own namespace and the cluster's default domain. This means that if there is a service named `foo` in Kubernetes namespace `bar`, then pods in the same namespace can access the service as `foo`, and pods in other namespaces can access the service as `foo.bar` Kubernetes supports a rich set of options for controlling DNS in different scenarios. You can read more about these in the Kubernetes guide [DNS for Services and Pods](https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/). ## NAT outgoing[​](#nat-outgoing) The Kubernetes network model specifies that pods must be able to communicate with each other directly using pod IP addresses. But it does not mandate that pod IP addresses are routable beyond the boundaries of the cluster. Many Kubernetes network implementations use [overlay networks](https://docs.tigera.io/calico-cloud/networking/training/about-networking#overlay-networks). Typically for these deployments, when a pod initiates a connection to an IP address outside of the cluster, the node hosting the pod will SNAT (Source Network Address Translation) map the source address of the packet from the pod IP to the node IP. This enables the connection to be routed across the rest of the network to the destination (because the node IP is routable). Return packets on the connection are automatically mapped back by the node replacing the node IP with the pod IP before forwarding the packet to the pod. When using Calico Cloud, depending on your environment, you can generally choose whether you prefer to run an overlay network, or prefer to have fully routable pod IPs. Calico Cloud also allows you to [configure outgoing NAT](https://docs.tigera.io/calico-cloud/networking/configuring/workloads-outside-cluster) for specific IP address ranges if more granularity is desired. ## Dual stack[​](#dual-stack) If you want to use a mix of IPv4 and IPv6 then you can enable Kubernetes [dual-stack](https://kubernetes.io/docs/concepts/services-networking/dual-stack/) mode. When enabled, all pods will be assigned both an IPv4 and IPv6 address, and Kubernetes Services can specify whether they should be exposed as IPv4 or IPv6 addresses. ## Additional resources[​](#additional-resources) - [The Kubernetes Network Model](https://kubernetes.io/docs/concepts/cluster-administration/networking/#the-kubernetes-network-model) - [Video: Everything you need to know about Kubernetes networking on AWS](https://www.projectcalico.org/everything-you-need-to-know-about-kubernetes-pod-networking-on-aws/) - [Video: Everything you need to know about Kubernetes networking on Azure](https://www.projectcalico.org/everything-you-need-to-know-about-kubernetes-networking-on-azure/) - [Video: Everything you need to know about Kubernetes networking on Google Cloud](https://www.projectcalico.org/everything-you-need-to-know-about-kubernetes-networking-on-google-cloud/) ### Configure Calico Cloud networking ## [📄️ Configure BGP peering](https://docs.tigera.io/calico-cloud/networking/configuring/bgp) [Configure BGP peering with full mesh, node-specific peering, ToR, and/or Calico route reflectors.](https://docs.tigera.io/calico-cloud/networking/configuring/bgp) ## [📄️ Configure BGP peering with nested clusters running on KubeVirt VMs](https://docs.tigera.io/calico-cloud/networking/configuring/bgp-to-workload) [Configure BGP peering with nested clusters running on KubeVirt VMs](https://docs.tigera.io/calico-cloud/networking/configuring/bgp-to-workload) ## [📄️ Deploy a dual ToR cluster](https://docs.tigera.io/calico-cloud/networking/configuring/dual-tor) [Configure a dual plane cluster for redundant connectivity between workloads.](https://docs.tigera.io/calico-cloud/networking/configuring/dual-tor) ## [📄️ Configure multiple Calico Cloud networks on a pod](https://docs.tigera.io/calico-cloud/networking/configuring/multiple-networks) [Configure a cluster with multiple Calico Cloud networks on each pod, and enforce security using Calico Cloud tiered network policy.](https://docs.tigera.io/calico-cloud/networking/configuring/multiple-networks) ## [📄️ Overlay networking](https://docs.tigera.io/calico-cloud/networking/configuring/vxlan-ipip) [Configure Calico to use IP in IP or VXLAN overlay networking so the underlying network doesn’t need to understand pod addresses.](https://docs.tigera.io/calico-cloud/networking/configuring/vxlan-ipip) ## [📄️ Advertise Kubernetes service IP addresses](https://docs.tigera.io/calico-cloud/networking/configuring/advertise-service-ips) [Configure Calico to advertise Kubernetes service cluster IPs and external IPs outside the cluster using BGP.](https://docs.tigera.io/calico-cloud/networking/configuring/advertise-service-ips) ## [📄️ Configure MTU to maximize network performance](https://docs.tigera.io/calico-cloud/networking/configuring/mtu) [Optimize network performance for workloads by configuring the MTU in Calico to best suit your underlying network.](https://docs.tigera.io/calico-cloud/networking/configuring/mtu) ## [📄️ Custom BGP Configuration](https://docs.tigera.io/calico-cloud/networking/configuring/custom-bgp-config) [Apply a custom BGP configuration](https://docs.tigera.io/calico-cloud/networking/configuring/custom-bgp-config) ## [📄️ Configure outgoing NAT](https://docs.tigera.io/calico-cloud/networking/configuring/workloads-outside-cluster) [Configure Calico Cloud networking to perform outbound NAT for connections from pods to outside of the cluster.](https://docs.tigera.io/calico-cloud/networking/configuring/workloads-outside-cluster) ## [📄️ Use a specific MAC address for a pod](https://docs.tigera.io/calico-cloud/networking/configuring/pod-mac-address) [Specify the MAC address for a pod instead of allowing the operating system to assign one](https://docs.tigera.io/calico-cloud/networking/configuring/pod-mac-address) ## [📄️ Use NodeLocal DNSCache in your cluster](https://docs.tigera.io/calico-cloud/networking/configuring/node-local-dns-cache) [Install NodeLocal DNSCache](https://docs.tigera.io/calico-cloud/networking/configuring/node-local-dns-cache) ## [📄️ Configure QoS Controls](https://docs.tigera.io/calico-cloud/networking/configuring/qos-controls) [Configure QoS (Quality of Service) Controls to limit ingress and/or egress bandwidth, packet rate and number of connections of Calico workloads.](https://docs.tigera.io/calico-cloud/networking/configuring/qos-controls) ### Configure BGP peering ## Big picture[​](#big-picture) Configure BGP (Border Gateway Protocol) between Calico nodes or peering with network infrastructure to distribute routing information. ## Value[​](#value) Calico Cloud nodes can exchange routing information over BGP to enable reachability for Calico Cloud networked workloads (Kubernetes pods or OpenStack VMs). In an on-premises deployment this allows you to make your workloads first-class citizens across the rest of your network. In public cloud deployments, it provides an efficient way of distributing routing information within your cluster, and is often used in conjunction with IPIP overlay or cross-subnet modes. ## Concepts[​](#concepts) ### BGP[​](#bgp) **BGP** is a standard protocol for exchanging routing information between routers in a network. Each router running BGP has one or more **BGP peers** - other routers which they are communicating with over BGP. You can think of Calico Cloud networking as providing a virtual router on each of your nodes. You can configure Calico Cloud nodes to peer with each other, with route reflectors, or with top-of-rack (ToR) routers. ### Common BGP topologies[​](#common-bgp-topologies) There are many ways to configure a BGP network depending on your environment. Here are some common ways it is done with Calico Cloud. ### Full-mesh[​](#full-mesh) When BGP is enabled, Calico Cloud’s default behavior is to create a **full-mesh** of internal BGP (iBGP) connections where each node peers with each other. This allows Calico Cloud to operate over any L2 network, whether public cloud or private cloud, or, if IPIP is [configured](https://docs.tigera.io/calico-cloud/networking/configuring/vxlan-ipip), to operate as an overlay over any network that does not block IPIP traffic. Calico Cloud does not use BGP for VXLAN overlays. > **SECONDARY:** Most public clouds support IPIP. The notable exception is Azure, which blocks IPIP traffic. So if you want to run Calico as an overlay network in Azure, you must [configure Calico Cloud to use VXLAN](https://docs.tigera.io/calico-cloud/networking/configuring/vxlan-ipip). Full-mesh works great for small and medium-size deployments of say 100 nodes or less, but at significantly larger scales full-mesh becomes less efficient, and we recommend using route reflectors. ### Route reflectors[​](#route-reflectors) To build large clusters of internal BGP (iBGP), **BGP route reflectors** can be used to reduce the number of BGP peerings used on each node. In this model, some nodes act as route reflectors and are configured to establish a full mesh amongst themselves. Other nodes are then configured to peer with a subset of those route reflectors (typically 2 for redundancy), reducing the total number BGP peering connections compared to full-mesh. ### Top of Rack (ToR)[​](#top-of-rack-tor) In **on-premises deployments**, you can configure Calico Cloud to peer directly with your physical network infrastructure. Typically, this involves disabling Calico Cloud’s default full-mesh behavior, and instead peer Calico Cloud with your L3 ToR routers. There are many ways to build an on-premises BGP network. How you configure your BGP is up to you - Calico Cloud works well with both iBGP and eBGP configurations, and you can effectively treat Calico Cloud like any other router in your network design. Depending on your topology, you may also consider using BGP route reflectors within each rack. However, this is typically needed only if the number of nodes in each L2 domain is large (> 100). For a deeper look at common on-premises deployment models, see [Calico over IP Fabrics](https://docs.tigera.io/calico-cloud/reference/architecture/design/l2-interconnect-fabric). ## Before you begin...[​](#before-you-begin) **Required** - Calico CNI ## How to[​](#how-to) > **SECONDARY:** Significantly changing Calico Cloud's BGP topology, such as changing from full-mesh to peering with ToRs, may result in temporary loss of pod network connectivity during the reconfiguration process. It is recommended to only make such changes during a maintenance window. - [Configure a global BGP peer](#configure-a-global-bgp-peer) - [Configure a per-node BGP peer](#configure-a-per-node-bgp-peer) - [Configure a node to act as a route reflector](#configure-a-node-to-act-as-a-route-reflector) - [Disable the default BGP node-to-node mesh](#disable-the-default-bgp-node-to-node-mesh) - [Change from node-to-node mesh to route reflectors without any traffic disruption](#change-from-node-to-node-mesh-to-route-reflectors-without-any-traffic-disruption) - [View BGP peering status for a node](#view-bgp-peering-status-for-a-node) - [View BGP info on all peers for a node](#view-bgp-info-on-all-peers-for-a-node) - [Change the default global AS number](#change-the-default-global-as-number) - [Change AS number for a particular node](#change-as-number-for-a-particular-node) - [Set a local AS number for a specific BGP peer](#set-a-local-as-number-for-a-specific-bgp-peer) - [Configure a BGP filter](#configure-a-bgp-filter) - [Configure a BGP peer with a BGP filter](#configure-a-bgp-peer-with-a-bgp-filter) ### Configure a global BGP peer[​](#configure-a-global-bgp-peer) Global BGP peers apply to all nodes in your cluster. This is useful if your network topology includes BGP speakers that will be peered with every Calico Cloud node in your deployment. The following example creates a global BGP peer that configures every Calico Cloud node to peer with **192.20.30.40** in AS **64567**. ```yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: my-global-peer spec: peerIP: 192.20.30.40 asNumber: 64567 ``` ### Configure a per-node BGP peer[​](#configure-a-per-node-bgp-peer) Per-node BGP peers apply to one or more nodes in the cluster. You can choose which nodes by specifying the node’s name exactly, or using a label selector. The following example creates a BGPPeer that configures every Calico Cloud node with the label, **rack: rack-1** to peer with **192.20.30.40** in AS **64567**. ```yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: rack1-tor spec: peerIP: 192.20.30.40 asNumber: 64567 nodeSelector: rack == 'rack-1' ``` ### Configure a node to act as a route reflector[​](#configure-a-node-to-act-as-a-route-reflector) Calico Cloud nodes can be configured to act as route reflectors. To do this, each node that you want to act as a route reflector must have a cluster ID - typically an unused IPv4 address. To configure a node to be a route reflector with cluster ID 244.0.0.1, run the following command. ```bash kubectl annotate node my-node projectcalico.org/RouteReflectorClusterID=244.0.0.1 ``` Typically, you will want to label this node to indicate that it is a route reflector, allowing it to be easily selected by a BGPPeer resource. You can do this with kubectl. For example: ```bash kubectl label node my-node route-reflector=true ``` Now it is easy to configure route reflector nodes to peer with each other and other non-route-reflector nodes using label selectors. For example: ```yaml kind: BGPPeer apiVersion: projectcalico.org/v3 metadata: name: peer-with-route-reflectors spec: nodeSelector: all() peerSelector: route-reflector == 'true' ``` > **SECONDARY:** Adding `routeReflectorClusterID` to a node spec will remove it from the node-to-node mesh immediately, tearing down the existing BGP sessions. Adding the BGP peering will bring up new BGP sessions. This will cause a short (about 2 seconds) disruption to data plane traffic of workloads running in the nodes where this happens. To avoid this, make sure no workloads are running on the nodes, by provisioning new nodes or by running `kubectl drain` on the node (which may itself cause a disruption as workloads are drained). ### Disable the default BGP node-to-node mesh[​](#disable-the-default-bgp-node-to-node-mesh) The default **node-to-node BGP mesh** may be turned off to enable other BGP topologies. To do this, modify the default **BGP configuration** resource. Run the following command to disable the BGP full-mesh: ```bash calicoctl patch bgpconfiguration default -p '{"spec": {"nodeToNodeMeshEnabled": false}}' ``` > **SECONDARY:** If the default BGP configuration resource does not exist, you need to create it first. See [BGP configuration](https://docs.tigera.io/calico-cloud/reference/resources/bgpconfig) for more information. > **SECONDARY:** Disabling the node-to-node mesh will break pod networking until/unless you configure replacement BGP peerings using BGPPeer resources. You may configure the BGPPeer resources before disabling the node-to-node mesh to avoid pod networking breakage. ### Change from node-to-node mesh to route reflectors without any traffic disruption[​](#change-from-node-to-node-mesh-to-route-reflectors-without-any-traffic-disruption) Switching from node-to-node BGP mesh to BGP route reflectors involves tearing down BGP sessions and bringing up new ones. This causes a short data plane network disruption (of about 2 seconds) for workloads running on the nodes in the cluster. To avoid this, you may provision route reflector nodes and bring their BGP sessions up before tearing down the node-to-node mesh sessions. Follow these steps to do so: 1. [Provision new nodes to be route reflectors.](#configure-a-node-to-act-as-a-route-reflector) The nodes [should not be schedulable](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) and they should have `routeReflectorClusterID` in their spec. These won't be part of the existing node-to-node BGP mesh, and will be the route reflectors when the mesh is disabled. These nodes should also have a label like `route-reflector` to select them for the BGP peerings. [Alternatively](https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/), you can drain workloads from existing nodes in your cluster by running `kubectl drain ` to configure them to be route reflectors, but this will cause a disruption on the workloads on those nodes as they are drained. 2. Also set up a [BGPPeer](#configure-a-node-to-act-as-a-route-reflector) spec to configure route reflector nodes to peer with each other and other non-route-reflector nodes using label selectors. 3. Wait for these peerings to be established. This can be [verified](#view-bgp-peering-status-for-a-node) by running `sudo calicoctl node status` on the nodes. Alternatively, you can create a [`CalicoNodeStatus` resource](https://docs.tigera.io/calico-cloud/reference/resources/caliconodestatus) to get BGP session status for the node. 4. [Disable the BGP node-to-node mesh for the cluster.](#disable-the-default-bgp-node-to-node-mesh) 5. If you did drain workloads from the nodes or created them as unschedulable, mark the nodes as schedulable again (e.g. by running `kubectl uncordon `). ### View BGP peering status for a node[​](#view-bgp-peering-status-for-a-node) Create a [CalicoNodeStatus resource](https://docs.tigera.io/calico-cloud/reference/resources/caliconodestatus) to monitor BGP session status for the node. Alternatively, you can run the `calicoctl node status` command on a given node to learn more about its BGP status. > **SECONDARY:** This command communicates with the local Calico Cloud agent, so you must execute it on the node whose status you are attempting to view. ### View BGP info on all peers for a node[​](#view-bgp-info-on-all-peers-for-a-node) You can use `calicoctl` to view the BGP information for all peers of a particular node, including connection status, routing statistics, and BGP state. This is useful for confirming that your configuration is behaving as desired, and for more detailed troubleshooting. Run the following command from anywhere you have access to `kubectl`: ```bash calicoctl bgp peers ``` Where `` is the resource name for one of the Calico node pods within your cluster. > **SECONDARY:** The above command can be run from anywhere you have access to kubectl. We recommend running it as a kubectl plugin. If you install the binary as a kubectl plugin using the above instructions, you can then run the command as follows: ```bash kubectl calico bgp peers ``` ### Change the default global AS number[​](#change-the-default-global-as-number) By default, all Calico nodes use the 64512 autonomous system, unless a per-node AS has been specified for the node. You can change the global default for all nodes by modifying the default **BGPConfiguration** resource. The following example command sets the global default AS number to **64513**. ```bash kubectl patch bgpconfiguration default -p '{"spec": {"asNumber": "64513"}}' ``` > **SECONDARY:** If the default BGP configuration resource does not exist, you need to create it first. See [BGP configuration](https://docs.tigera.io/calico-cloud/reference/resources/bgpconfig) for more information. ### Change AS number for a particular node[​](#change-as-number-for-a-particular-node) You can configure an AS for a particular node by modifying the node object using `calicoctl`. For example, the following command changes the node named **node-1** to belong to **AS 64514**. ```bash calicoctl patch node node-1 -p '{"spec": {"bgp": {"asNumber": "64514"}}}' ``` ### Set a local AS number for a specific BGP peer[​](#set-a-local-as-number-for-a-specific-bgp-peer) You can configure a distinct local AS number for an individual BGP peer by modifying the `BGPPeer` resource using `calicoctl` or `kubectl`. This overrides the node-wide or global AS number defined in the `BGPConfiguration`, allowing different local ASNs to be used on a per-peer basis. If `localASNumber` is not set, Calico will fall back to the node-wide or global AS number. ```bash kubectl patch bgppeer -p '{"spec": {"localASNumber": 64577}}' ``` ### Configure a BGP filter[​](#configure-a-bgp-filter) BGP filters control which routes are imported and exported between BGP peers. The BGP filter rules (importVX, exportVX) are applied sequentially, taking the `action` of the first matching rule. When no rules are matched, the default `action` is `Accept`. In order for a BGPFilter to be used in a BGP peering, its `name` must be added to `filters` of the corresponding BGPPeer resource. The following example creates a BGPFilter ```yaml apiVersion: projectcalico.org/v3 kind: BGPFilter metadata: name: my-filter spec: exportV4: - action: Accept matchOperator: In cidr: 77.0.0.0/16 - action: Reject source: RemotePeers - action: Reject interface: '*.calico' importV4: - action: Reject matchOperator: NotIn cidr: 44.0.0.0/16 exportV6: - action: Reject source: RemotePeers - action: Reject interface: '*.calico' importV6: - action: Accept matchOperator: Equal cidr: 5000::0/64 - action: Reject ``` ### Configure a BGP peer with a BGP filter[​](#configure-a-bgp-peer-with-a-bgp-filter) BGP peers can use BGP filters to control which routes are imported or exported between them. The following example creates a BGPFilter and associates it with a BGPPeer > **SECONDARY:** BGPFilters are applied in the order listed on a BGPPeer ```yaml kind: BGPFilter apiVersion: projectcalico.org/v3 metadata: name: first-bgp-filter spec: exportV4: - action: Accept matchOperator: In cidr: 77.0.0.0/16 source: RemotePeers importV4: - action: Reject matchOperator: NotIn cidr: 44.0.0.0/16 exportV6: - action: Reject interface: '*.calico' importV6: - action: Accept matchOperator: Equal cidr: 5000::0/64 --- kind: BGPFilter apiVersion: projectcalico.org/v3 metadata: name: second-bgp-filter spec: exportV4: - action: Accept matchOperator: In cidr: 77.0.0.0/16 interface: '*.calico' importV4: - action: Reject matchOperator: NotIn cidr: 44.0.0.0/16 exportV6: - action: Reject source: RemotePeers importV6: - action: Reject --- kind: BGPPeer apiVersion: projectcalico.org/v3 metadata: name: peer-with-filter spec: peerSelector: has(filter-bgp) filters: - first-bgp-filter - second-bgp-filter ``` ## Additional resources[​](#additional-resources) - [Node resource](https://docs.tigera.io/calico-cloud/reference/resources/node) - [Configure BGP peering with nested clusters running on Kubevirt VMs](https://docs.tigera.io/calico-cloud/networking/configuring/bgp-to-workload) - [BGP configuration resource](https://docs.tigera.io/calico-cloud/reference/resources/bgpconfig) - [BGP peer resource](https://docs.tigera.io/calico-cloud/reference/resources/bgppeer) - [BGP filter resource](https://docs.tigera.io/calico-cloud/reference/resources/bgpfilter) ### Configure BGP peering with nested clusters running on KubeVirt VMs ## Big picture[​](#big-picture) Configure BGP peering with Calico Cloud nodes running on KubeVirt VMs that are running on your Kubernetes cluster. ## Value[​](#value) Run nested Kubernetes clusters in VMs on your parent Kubernetes cluster. Avoid network overlay overheads; extend layer 3 network reachability all the way from pods running in the nested cluster to your physical network infrastructure. ## Concepts[​](#concepts) This guide assumes you have a good understanding of BGP and Calico Cloud's BGP peering options. The guide makes use of several Calico Cloud resources, including: - [BGPPeer](https://docs.tigera.io/calico-cloud/reference/resources/bgppeer) - Used to configure peerings both with external BGP routers and with BGP-enabled workloads. - [BGPFilter](https://docs.tigera.io/calico-cloud/reference/resources/bgpfilter) - Defines a BGP filter to control which routes are imported or exported between BGP peers. - [BGPConfiguration](https://docs.tigera.io/calico-cloud/reference/resources/bgpconfig) - Configures global BGP settings for the cluster. See [Configure BGP peering](https://docs.tigera.io/calico-cloud/networking/configuring/bgp) for a general overview of these concepts. ## Supported BGP topologies[​](#supported-bgp-topologies) ![Diagram showing supported BGP topologies for nested clusters running on KubeVirt VMs, with eBGP peering between parent and nested clusters](https://docs.tigera.io/assets/images/nested-bgp-caa4c05c0d1dd680a3d7528c95e1dac7.svg) Both parent and nested clusters must be configured to use non-overlay BGP networking. eBGP must be used both for peering between parent cluster and top-of-rack router and for peering between parent cluster and nested workload cluster. This means that the AS numbers of the parent cluster, the nested cluster, and the top-of-rack router must all be different. iBGP peerings to workloads or ToR routers are not supported: - iBGP to the ToR router is not supported because there is currently no way to configure "next hop self" behaviour on that peering (so the ToR would receive the nested cluster's routes with the workload's IP as next hop, rather than the parent cluster node's IP). - iBGP to the nested cluster has not been tested. It may work with appropriate `BGPFilter` resources to ensure the correct routes are re-advertised. Peering with workloads and using a BGP mesh within the cluster is not supported; this combination has not been tested and it is likely to require additional `BGPFilter` resources to ensure routes are correctly re-advertised. ## Supported KubeVirt networking modes[​](#supported-kubevirt-networking-modes) Only KubeVirt's "bridged" networking mode is supported. This is because the peering from parent cluster to the workload is done using the "pod IP" of the workload. This must agree with the IP that the workload itself uses to source traffic. ## How to[​](#how-to) - [Configure global BGP settings](#configure-global-bgp-settings) - [Create BGPFilter resources](#configure-bgpfilter-resources) - [Configure parent cluster BGP topology](#configure-parent-cluster-bgp-topology) - [Configure BGP peering with workloads](#configure-bgp-peering-with-workloads) - [Configure more than one nested cluster](#configure-more-than-one-nested-cluster) ### Configure global BGP settings[​](#configure-global-bgp-settings) In order to ease configuration of the `BGPPeer` resources in the nested clusters, the workload BGP peering feature uses a single virtual IP (per IP version) that is provisioned on every node in the parent cluster. The workloads are configured to peer with this virtual IP rather than needing to know the specific IP of their parent node. To avoid conflicts with your network, the addresses are configurable through the `BGPConfiguration` resource, and there are no defaults. They must be configured to enable the feature: For IPv4, we recommend using a link-local IP address that is not otherwise in use on the parent cluster's network. Note that Calico Cloud uses 169.254.0.1 and 169.254.0.2 internally so these must not be used. You may also wish to avoid cloud provider metadata addresses, such as 169.254.169.254 (as used by AWS). In this guide we'll use 169.254.0.179; you can use the same address unless it conflicts with some other use on your nodes. If using the IPv6 address, you *must not* use a link local address (because IPv6 requires link-local addresses to be qualified with a scope tag, and Calico Cloud currently has no way to provision the correct scopes). Instead, we recommend using a ULA address that is not otherwise in use in your network. In this guide we'll use fdc9:9723:09bc::1 (which was chosen randomly according to RFC 4193 for this guide, so it may be suitable for your use as well). Once you have chosen the IP address(es), configure the `BGPConfiguration` resource. (You can omit the IPv6 address if not using IPv6.) ```yaml apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: ... localWorkloadPeeringIPV4: "169.254.0.179" localWorkloadPeeringIPV6: "fdc9:9723:09bc::1" ``` ### Create `BGPFilter` resources[​](#create-bgpfilter-resources) Before creating the `BGPPeer` resource to peer with the workloads, we create `BGPFilter` resources to control which routes are imported and exported. - Because the workloads are running on the Calico Cloud cluster, we can assume that they already have a valid default route via their host node. Hence, we don't need to export any routes to the workloads. Create the following `BGPFilter` resource, which can be used to block export of any routes. ```yaml apiVersion: projectcalico.org/v3 kind: BGPFilter metadata: name: no-export spec: exportV4: - action: Reject exportV6: - action: Reject ``` - For added security, it is also wise to limit what routes we'll accept from the workloads. Create a `BGPFilter` resource that only accepts routes that are within the IP pool of the nested cluster. For example, if the nested cluster's IP pools are 10.123.0.0/16 and ca11:c0::/48: ```yaml apiVersion: projectcalico.org/v3 kind: BGPFilter metadata: name: accept-nested-ip-pools spec: importV4: - action: Accept matchOperator: In cidr: 10.123.0.0/16 # IP pool CIDR of nested cluster importV6: - action: Accept matchOperator: In cidr: ca11:c0::/48 # IP pool CIDR of nested cluster ``` In addition we need to create a `BGPFilter` to tell Calico Cloud to re-advertise routes learned from the workloads to the ToR. Create the following `BGPFilter` resource, adjusting the CIDRs as appropriate for your nested cluster: ```yaml apiVersion: projectcalico.org/v3 kind: BGPFilter metadata: name: export-to-tor spec: exportV4: - action: Accept matchOperator: In cidr: 10.123.0.0/16 source: RemotePeers exportV6: - action: Accept matchOperator: In cidr: ca11:c0::/48 source: RemotePeers ``` ### Configure parent cluster BGP topology[​](#configure-parent-cluster-bgp-topology) Configure the parent cluster to use BGP networking and to peer with its top-of-rack router(s) over eBGP. For example, by using the [downward default model](https://docs.tigera.io/calico-cloud/reference/architecture/design/l3-interconnect-fabric#the-downward-default-model). Include the `BGPFilter` resource defined above in the `BGPPeer` resource used to peer with the ToR: ```yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: node-tor-peer spec: peerIP: asNumber: filters: - export-to-tor ``` ### Configure BGP peering with workloads[​](#configure-bgp-peering-with-workloads) In the parent cluster, create a `BGPPeer` resource to peer with the workloads. - The `localWorkloadSelector` field is used to select which workloads to peer with. In this example, we select workloads with the label `color=red`. - Include the `BGPFilter` resources defined above in the `BGPPeer` resource to control which routes are imported and exported. ```yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: peer-to-workloads spec: localWorkloadSelector: color == 'red' asNumber: filters: - no-export - accept-nested-ip-pools ``` Both per-node and global `BGPPeer` resources are supported. To limit the peering to certain nodes, use the `nodeSelector` field. > **SECONDARY:** `localWorkloadSelector` is scoped to all workloads and it does not currently have a corresponding "namespace selector". To match on namespace labels, you can prefix the label name with `pcns.`. In the child cluster, - Ensure that the node-to-node mesh is disabled in the `BGPConfiguration` resource and that the workload cluster's AS number is set: ```yaml apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: nodeToNodeMeshEnabled: false asNumber: ``` - Create a `BGPPeer` resource (per desired IP version) that peers with the parent clusters' virtual IP: ```yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: peer-to-parent-cluster spec: peerIP: 169.254.0.179 asNumber: --- apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: peer-to-parent-cluster-v6 spec: peerIP: "fdc9:9723:09bc::1" asNumber: ``` ### Configure more than one nested cluster[​](#configure-more-than-one-nested-cluster) You can configure more than one nested cluster by: - Choosing a unique non-overlapping IP pool for each nested cluster. - In the parent cluster: creating a new import `BGPFilter` resource for each cluster to import its routes. - In the parent cluster: creating a new, uniquely-named `BGPPeer` resource for each cluster to peer with the parent cluster. You can re-use the shared "no-export" `BGPFilter` resource with every nested cluster peering. - In the parent cluster: updating the `export-to-tor` `BGPFilter` resource to include the new cluster's CIDRs. - In the nested cluster: adding a copy of the `peer-to-parent-cluster` `BGPPeer`(s). These should be the same in all nested clusters since a parent cluster always uses the same virtual IP for all nested clusters. > **SECONDARY:** If you plan to use many nested clusters consider allocating their IP pools from a single larger CIDR. Then the `export-to-tor` filter can be simplified to match on the larger CIDR only. > > If you trust the nested clusters, similarly, you could also re-use a single `accept-nested-ip-pools` filter for all child clusters. ## Additional resources[​](#additional-resources) - [BGP guide](https://docs.tigera.io/calico-cloud/networking/configuring/bgp) - [BGP L3 interconnect guide](https://docs.tigera.io/calico-cloud/reference/architecture/design/l3-interconnect-fabric) - [Node resource](https://docs.tigera.io/calico-cloud/reference/resources/node) - [BGP configuration resource](https://docs.tigera.io/calico-cloud/reference/resources/bgpconfig) - [BGP peer resource](https://docs.tigera.io/calico-cloud/reference/resources/bgppeer) - [BGP filter resource](https://docs.tigera.io/calico-cloud/reference/resources/bgpfilter) ### Deploy a dual ToR cluster ## Big picture[​](#big-picture) Deploy a dual plane cluster to provide redundant connectivity between your workloads for on-premises deployments. > **SECONDARY:** Dual ToR is not supported if you are using BGP with encapsulation (VXLAN or IP-in-IP). ## Value[​](#value) A dual plane cluster provides two independent planes of connectivity between all cluster nodes. If a link or software component breaks somewhere in one of those planes, cluster nodes can still communicate over the other plane, and the cluster as a whole continues to operate normally. ## Concepts[​](#concepts) ### Dual plane connectivity, aka "dual ToR"[​](#dual-plane-connectivity-aka-dual-tor) Large on-prem Kubernetes clusters, split across multiple server racks, can use two or more independent planes of connectivity between all the racks. The advantages are: - The cluster can still function, even if there is a single break in connectivity somewhere. - The cluster can load balance across the bandwidth of *both* planes, when both planes are available. The redundant approach can be applied within each rack as well, such that each node has two or more independent connections to those connectivity planes. Typically, each rack has two top-of-rack routers ("ToRs") and each node has two fabric-facing interfaces, each of which connects over a separate link or Ethernet to one of the ToRs for the rack. Here's an example of how a dual plane setup might look, with just two racks and two nodes in each rack. For simplicity, we've shown the connections *between* racks as single links; in reality that would be more complex, but still following the overall dual plane paradigm. ![dual-tor](https://docs.tigera.io/assets/images/dual-tor-3822091b69ea7dab14d53fa934356e68.png) Because of the two ToRs per rack, the whole setup is often referred to as "dual ToR". ### Network design for a dual ToR cluster[​](#network-design-for-a-dual-tor-cluster) For a dual ToR cluster to operate seamlessly when there is a break on one of the planes, several things are needed. - Each node should have a stable IP address that is independent of its per-interface addresses and remains valid if the connectivity through *one* of those interfaces goes down. - Each node must somehow know or learn the stable IP address of every other node. - Wherever a connection (other than BGP) is to or from a *node* (as opposed to a non-host-networked pod), that connection should use the node's stable address as its destination or source IP (respectively), so that the connection can continue working if one of the planes has an outage. - Importantly, this includes connections that Kubernetes uses as part of its own control plane, such as between the Kubernetes API server and kubelet on each node. Ideally, therefore, the stable IP address setup on each node should happen before Kubernetes starts running. - BGP is an exception to the previous points - in fact, the *only* exception - because we want each node's BGP peerings to be interface-specific and to reflect what is actually reachable, moment by moment, over that interface. The Linux routing table then automatically adjusts so that the route to each remote destination is either ECMP - when both planes are up - or non-ECMP when it can only be reached over one of the planes. - BGP peerings should be configured to detect any outages, and to propagate their consequences, as quickly as possible, so that the routing can quickly respond on each node. Note that this is quite different from the reasoning for a single connectivity plane, where it's better to delay any network churn, on the assumption that an outage will be quickly fixed. Finally, to spread load evenly and maximise use of both planes, when both available, the routers and Linux kernel need to be configured for efficient ECMP. ### Calico's early networking architecture[​](#calicos-early-networking-architecture) Calico Cloud's cnx-node image can be run in an "early networking" mode, on each node, to perform all of the above points that are needed before Kubernetes starts running. That means that it: - Provisions the stable IP address. - Makes the changes needed to ensure that the stable address will be used as the source IP for any outgoing connections from the node. - Starts running BGP, peering with the node's ToRs, to advertise the node's stable address to other nodes. - Configures efficient ECMP in the Linux kernel (with `fib_multipath_hash_policy=1` and `fib_multipath_use_neigh=1`). More detail is given below on how to run this early networking image. A key point is that it must run as soon as possible after each node boot, and before Kubernetes starts on the node, so it is typically run as a Docker or podman container. After its start-of-day provisioning, the early networking container keeps running so that it can tag-team the BGP role with Calico's regular BGP service running inside the cnx-node *pod*: - Initially the cnx-node pod does not yet exist, so the early networking BGP runs to advertise out the node's stable address. - After Kubernetes has started on the node, and Calico has been installed in Kubernetes, the cnx-node pod runs and starts its own BGP service. The early networking container spots that the regular BGP service is now running and so shuts down its own BGP. Now the regular BGP service handles the advertisement of the stable address, as well as pod IPs and so on. - Later, the cnx-node pod might be shut down, e.g. for restart or upgrade. If the downtime continues for longer than the graceful restart period, the early networking container spots this and restarts its own BGP, to ensure that the node's stable IP address continues to be advertised to other nodes. The cycle can now repeat from the "Initially" state above. > **SECONDARY:** The default graceful restart period is 120s for traditional BGP GR and 3600s for LLGR. ### BGP configuration for rapid outage detection[​](#bgp-configuration-for-rapid-outage-detection) A dual ToR cluster needs Calico BGPPeer resources to specify how each node should peer with its ToRs. The remaining parts of the dual ToR network design are implemented as properties of those BGP peerings, and as corresponding properties on the BGP configuration between and within the ToRs and core infrastructure. Specifically, on Calico's BGPPeer resource, - the `failureDetectionMode` field is used to enable BFD - the `restartMode` field can be used to enable long-lived graceful restart (LLGR). See below for more on the benefits of these settings. When they are used, consistent settings are needed on the ToRs and core infrastructure. ### ECMP routing[​](#ecmp-routing) An "Equal Cost Multiple Path" (ECMP) route is one that has multiple possible ways to reach a given destination or prefix, all of which are considered to be equally good. A dual ToR setup naturally generates ECMP routes, with the different paths going over the different connectivity planes. When using an ECMP route, Linux decides how to balance traffic across the available paths, including whether this is informed by TCP and UDP port numbers as well as source and destination IP addresses, whether the decision is made per-packet, per-connection, or in some other way, and so on; and the details here have varied with Linux kernel version. For a clear account of the exact options and behaviors for different kernel versions, please see [this blog](https://web.archive.org/web/20210204031636/https://cumulusnetworks.com/blog/celebrating-ecmp-part-two/). ### BFD[​](#bfd) Bidirectional Forwarding Detection (BFD) is [a protocol](https://tools.ietf.org/html/rfc5880) that detects very quickly when forwarding along a particular path stops working - whether that's because a link has broken somewhere, or some software component along the path. In a dual ToR setup, rapid failure detection is important so that traffic flows within the cluster can quickly adjust to using the other available connectivity plane. ### Long lived graceful restart[​](#long-lived-graceful-restart) Long Lived Graceful Restart (LLGR) is [an extension for BGP](https://tools.ietf.org/html/draft-uttaro-idr-bgp-persistence-05) that handles link failure by lowering the preference of routes over that link. This is a compromise between the base BGP behaviour - which is immediately to remove those routes - and traditional BGP Graceful Restart behaviour - which is not to change those routes at all, until some configured time has passed. For a dual ToR setup, LLGR is helpful, as explained in more detail by [this blog](https://vincent.bernat.ch/en/blog/2018-bgp-llgr) , because: - If a link fails somewhere, the immediate preference lowering allows traffic to adjust immediately to use the other connectivity plane. - If a node is restarted, we still get the traditional Graceful Restart behaviour whereby routes to that node persist in the rest of the network. ### Default routing and "nearly default" routes[​](#default-routing-and-nearly-default-routes) Calico's early networking architecture - and more generally, the considerations for dual ToR that are presented on this page - is compatible with many possible [L3 fabric designs](https://docs.tigera.io/calico-cloud/reference/architecture/design/l3-interconnect-fabric) . One of the options in such designs is "downward default", which means that each ToR only advertises the default route to its directly connected nodes, even when it has much more detailed routing information. "Downward default" works because the ToR should indeed be the node's next hop for all destinations, except for directly connected nodes in the same rack. In a dual ToR cluster, each node has two ToRs, and "downward default" should result in the node having an ECMP default route like this: ```text default proto bird nexthop via 172.31.11.100 dev eth0 nexthop via 172.31.12.100 dev eth0 ``` If one of the planes is broken, BGP detects and propagates the outage and that route automatically changes to a non-ECMP route via the working plane: ```text default via 172.31.12.100 dev eth0 proto bird ``` That is exactly the behaviour that is wanted in a dual ToR cluster. The snag with it is that there can be other procedures in the node's operating system that also update the default route - in particular, DHCP - and that can interfere with this desired behaviour. For example, if a DHCP lease renewal occurs for one of the node's interfaces, the node may then replace the default route as non-ECMP via that interface. A simple way to avoid such interference is to export the "nearly default" routes 0.0.0.0/1 and 128.0.0.0/1 from the ToRs, instead of the true default route 0.0.0.0/0. 0.0.0.0/1 and 128.0.0.0/1 together cover the entire IPv4 address space and so provide correct dual ToR routing for any destination outside the local rack. They also mask the true default route 0.0.0.0/0, by virtue of having longer prefixes (1 bit instead of 0 bits), and so it no longer matters if there is any other programming of the true default route on the node. ## Before you begin[​](#before-you-begin) **Unsupported** - AKS - EKS - GKE **Required** - Calico CNI ## How to[​](#how-to) - [Prepare YAML resources describing the layout of your cluster](#prepare-yaml-resources-describing-the-layout-of-your-cluster) - [Arrange for dual-homed nodes to run cnx-node on each boot](#arrange-for-dual-homed-nodes-to-run-cnx-node-on-each-boot) - [Configure your ToR routers and infrastructure](#configure-your-tor-routers-and-infrastructure) - [Install Kubernetes and Calico Cloud](#install-kubernetes-and-calico-enterprise) - [Verify the deployment](#verify-the-deployment) ### Prepare YAML resources describing the layout of your cluster[​](#prepare-yaml-resources-describing-the-layout-of-your-cluster) 1. Prepare BGPPeer resources to specify how each node in your cluster should peer with the ToR routers in its rack. For example, if your rack 'A' has ToRs with IPs 172.31.11.100 and 172.31.12.100 and the rack AS number is 65001: ```yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: ra1 spec: nodeSelector: "rack == 'ra' || rack == 'ra_single'" peerIP: 172.31.11.100 asNumber: 65001 sourceAddress: None --- apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: ra2 spec: nodeSelector: "rack == 'ra'" peerIP: 172.31.12.100 asNumber: 65001 sourceAddress: None ``` > **SECONDARY:** The effect of the `nodeSelector` fields here is that any node with label `rack: ra` will peer with both these ToRs, while any node with label `rack: ra_single` will peer with only the first ToR. For optimal dual ToR function and resilience, nodes in rack 'A' should be labelled `rack: ra`, but `rack: ra_single` can be used instead on any nodes which cannot be dual-homed. Repeat for as many racks as there are in your cluster. Each rack needs a new pair of BGPPeer resources with its own ToR addresses and AS number, and `nodeSelector` fields matching the nodes that should peer with its ToR routers. Depending on what your ToR supports, consider also setting these fields in each BGPPeer: - `failureDetectionMode: BFDIfDirectlyConnected` to enable BFD, when possible, for fast failure detection. > **SECONDARY:** Calico Cloud only supports BFD on directly connected peerings, but in practice nodes are normally directly connected to their ToRs. - `restartMode: LongLivedGracefulRestart` to enable LLGR handling when the node needs to be restarted, if your ToR routers support LLGR. If not, we recommend instead `maxRestartTime: 10s`. - `birdGatewayMode: DirectIfDirectlyConnected` to enable the "direct" next hop algorithm, if that is helpful for optimal interworking with your ToR routers. > **SECONDARY:** For directly connected BGP peerings, BIRD provides two gateway computation modes, "direct" and "recursive". > > "recursive" is the default, but "direct" can give better results when the ToR also acts as the route reflector (RR) for the rack. Specifically, a combined ToR/RR should ideally keep the BGP next hop intact (aka "next hop keep") when reflecting routes from other nodes in the same rack, but add itself as the BGP next hop (aka "next hop self") when forwarding routes from outside the rack. If your ToRs can be configured to do that, fine. If they cannot, an effective workaround is to configure the ToRs to do "next hop keep" for all routes, with "gateway direct" on the Calico Cloud nodes. In effect the “gateway direct” applies a “next hop self” when needed, but otherwise not. 2. Prepare this BGPConfiguration resource to [disable the full node-to-node mesh](https://docs.tigera.io/calico-cloud/networking/configuring/bgp#disable-the-default-bgp-node-to-node-mesh): ```yaml apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: nodeToNodeMeshEnabled: false ``` 3. Prepare disabled IPPool resources for the CIDRs from which you will allocate stable addresses for dual-homed nodes. For example, if the nodes in rack 'A' will have stable addresses from 172.31.10.0/24: ```yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: ra-stable spec: cidr: 172.31.10.0/24 disabled: true nodeSelector: all() ``` If the next rack uses a different CIDR, define a similar IPPool for that rack, and so on. > **SECONDARY:** These IPPool definitions tell Calico Cloud's BGP component to export routes within the given CIDRs, which is essential for the core BGP infrastructure to learn how to route to each stable address. `disabled: true` tells Calico Cloud *not* to use these CIDRs for pod IPs. 4. Prepare an enabled IPPool resource for your default CIDR for pod IPs. For example: ```yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: default-ipv4 spec: cidr: 10.244.0.0/16 nodeSelector: all() ``` > **SECONDARY:** The CIDR must match what you specify elsewhere in the Kubernetes installation. For example, `networking.clusterNetwork.cidr` in OpenShift's install config, or `--pod-network-cidr` with kubeadm. You should not specify `ipipMode` or `vxlanMode`, as these are incompatible with dual ToR operation. `natOutgoing` can be omitted, as here, if your core infrastructure will perform an SNAT for traffic from pods to the Internet. 5. Prepare an EarlyNetworkConfiguration resource to specify the additional information that is needed for each node in a multi-rack dual ToR cluster: - The stable address for the node. - Its BGP AS number. - The IPs that the node should peer with, when cnx-node runs as a container for early networking setup after each node boot. - Any labels that the node should have, so as to match the right BGPPeer definitions for its rack, when cnx-node runs as a Kubernetes pod. With OpenShift, also add a toplevel `platform: openshift` setting. > **SECONDARY:** `platform: openshift` triggers additional per-node setup that is needed during OpenShift's bootstrapping phase. For example, with IP addresses and AS numbers similar as for other resources above: ```yaml apiVersion: projectcalico.org/v3 kind: EarlyNetworkConfiguration spec: platform: openshift nodes: # worker1 - interfaceAddresses: - 172.31.11.3 - 172.31.12.3 stableAddress: address: 172.31.10.3 asNumber: 65001 peerings: - peerIP: 172.31.11.100 - peerIP: 172.31.12.100 labels: rack: ra # worker2 - interfaceAddresses: - 172.31.21.4 - 172.31.22.4 stableAddress: address: 172.31.20.4 asNumber: 65002 peerings: - peerIP: 172.31.21.100 - peerIP: 172.31.22.100 labels: rack: rb ... ``` 6. Prepare a ConfigMap resource named "bgp-layout", in namespace "tigera-operator", that wraps the EarlyNetworkConfiguration like this: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: bgp-layout namespace: tigera-operator data: earlyNetworkConfiguration: | apiVersion: projectcalico.org/v3 kind: EarlyNetworkConfiguration spec: nodes: # worker1 - interfaceAddresses: ... ``` > **SECONDARY:** EarlyNetworkConfiguration supplies labels and AS numbers to apply to each Calico node, as well as peering and other network configuration to use during node startup prior to receiving BGPPeer and BGPConfiguration resources from the datastore. EarlyNetworkConfiguration will be superseded by any BGPPeer or BGPConfiguration resources after successful startup. ### Arrange for dual-homed nodes to run cnx-node on each boot[​](#arrange-for-dual-homed-nodes-to-run-cnx-node-on-each-boot) Calico Cloud's cnx-node image normally runs as a Kubernetes pod, but for dual ToR setup it must also run as a container after each boot of a dual-homed node. For example: ```text podman run --privileged --net=host \ -v /calico-early:/calico-early -e CALICO_EARLY_NETWORKING=/calico-early/cfg.yaml \ quay.io/tigera/cnx-node:latest ``` The environment variable `CALICO_EARLY_NETWORKING` must point to the EarlyNetworkConfiguration prepared above, so that EarlyNetworkConfiguration YAML must be copied into a file on the node (here, `/calico-early/cfg.yaml`) and mapped into the cnx-node container. We recommend defining systemd services to ensure that early networking runs on each boot, and before kubelet starts on the node. Following is an example that may need tweaking for your particular platform, but that illustrates the important points. Firstly, a "calico-early" service that runs the Calico early networking on each boot: ```text [Unit] Wants=network-online.target After=network-online.target After=nodeip-configuration.service [Service] ExecStartPre=/bin/sh -c "rm -f /etc/systemd/system/kubelet.service.d/20-nodenet.conf /etc/systemd/system/crio.service.d/20-nodenet.conf; systemctl daemon-reload" ExecStartPre=-/bin/podman rm -f calico-early ExecStartPre=/bin/mkdir -p /etc/calico-early ExecStartPre=/bin/sh -c "test -f /etc/calico-early/details.yaml || /bin/curl -o /etc/calico-early/details.yaml http://172.31.1.1:8080/calico-early/details.yaml" ExecStart=/bin/podman run --rm --privileged --net=host --name=calico-early -v /etc/calico-early:/etc/calico-early -e CALICO_EARLY_NETWORKING=/etc/calico-early/details.yaml quay.io/tigera/cnx-node:latest [Install] WantedBy=multi-user.target ``` > **SECONDARY:** > > - You must also install your Tigera-issued pull secret at `/root/.docker/config.json`, on each node, to enable pulling from quay.io/. > - Some OpenShift versions have a `nodeip-configuration` service that configures kubelet's `--node-ip` option **wrongly** for a dual ToR setup. The `After=nodeip-configuration.service` setting and the deletion of `20-nodenet.conf` undo that service's work so that kubelet can choose its own IP correctly (using a reverse DNS lookup). > - The `/bin/curl ...` line shows how you can download the EarlyNetworkConfiguration YAML from a central hosting point within your cluster. Secondly, a "calico-early-wait" service that delays kubelet until after the Calico early networking setup is in place: ```text [Unit] After=calico-early.service Before=kubelet.service [Service] Type=oneshot ExecStart=/bin/sh -c "while sleep 5; do grep -q 00000000:1FF3 /proc/net/tcp && break; done; sleep 15" [Install] WantedBy=multi-user.target ``` > **SECONDARY:** > > - The `ExecStart` line here arranges that kubelet will not start running until the calico-early service has started listening on port 8179 (hex `1FF3`). 8179 is the port that the calico-early service uses for pre-Kubernetes BGP. > - We have sometimes observed issues if kubelet starts immediately after Calico's early networking setup, because of NetworkManager toggling the hostname. The final `sleep 15` allows for such changes to settle down before kubelet starts. On OpenShift you should wrap the above service definitions in `MachineConfig` resources for the control and worker nodes. On other platforms either define and enable the above services directly, or use whatever API the platform provides to define and enable services on new nodes. ### Configure your ToR routers and infrastructure[​](#configure-your-tor-routers-and-infrastructure) You should configure your ToR routers to accept all the BGP peerings from Calico Cloud nodes, to reflect routes between the nodes in that rack, and to propagate routes between the ToR routers in different racks. In addition we recommend consideration of the following points. BFD should be enabled if possible on all BGP sessions - both to the Calico Cloud nodes, and between racks in your core infrastructure - so that a break in connectivity anywhere can be rapidly detected. The handling should be to initiate LLGR procedures if possible, or else terminate the BGP session non-gracefully. LLGR should be enabled if possible on all BGP sessions - again, both to the Calico Cloud nodes, and between racks in your core infrastructure. Traditional BGP graceful restart should not be used, because this will delay the cluster's response to a break in one of the connectivity planes. ### Install Kubernetes and Calico Cloud[​](#install-kubernetes-and-calico-cloud) Details here vary, depending on **when** your Kubernetes installer gives an opportunity for you to define custom resources, but fundamentally what is needed here is to perform the installation as usual, except that all of the Calico resources prepared above, except the EarlyNetworkConfiguration, must be added into the datastore **before** the cnx-node pods start running on any node. We can illustrate this by looking at two examples: with OpenShift, and when adding Calico to an existing Kubernetes cluster. **OpenShift** With OpenShift, follow our documentation as far as the option to provide additional configuration. Then use `kubectl create configmap ...`, as that documentation says, to combine the prepared BGPPeer, BGPConfiguration and IPPool resources into a `calico-resources` ConfigMap. Place the generated file in the manifests directory for the OpenShift install. Also place the "bgp-layout" ConfigMap file in the manifests directory. Now continue with the OpenShift install process, and it will take care of adding those resources into the datastore as early as possible. **Adding to an existing Kubernetes cluster** Follow our documentation as far as the option for installing any custom Calico resources. Then use `calicoctl`, as that documentation says, to install the prepared BGPPeer, BGPConfiguration and IPPool resources. Also use `kubectl` to install the "bgp-layout" ConfigMap. Now continue with the Calico Cloud install process, and you should observe each node establishing BGP sessions with its ToRs. ### Verify the deployment[​](#verify-the-deployment) If you examine traffic and connections within the cluster - for example, using `ss` or `tcpdump` - you should see that all connections use loopback IP addresses or pod CIDR IPs as their source and destination. For example: - The kubelet on each node connecting to the API server. - The API server's connection to its backing etcd database, and peer connections between the etcd cluster members. - Pod connections that involve an SNAT or MASQUERADE in the data path, as can be the case when connecting to a Service through a cluster IP or NodePort. At the point of the SNAT or MASQUERADE, a loopback IP address should be used. - Direct connections between pod IPs on different nodes. The only connections using interface-specific addresses should be BGP. > **SECONDARY:** If you plan to use [Egress Gateways](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-on-prem) in your cluster, you must also adjust the cnx-node IP auto-detection method to pick up the stable IP, for example using the `interface: lo` setting (The default first-found setting skips over the lo interface). This can be configured via the Calico Cloud [Installation resource](https://docs.tigera.io/calico-cloud/reference/installation/api#nodeaddressautodetection). If you look at the Linux routing table on any cluster node, you should see ECMP routes like this to the loopback address of every other node in other racks: ```text 172.31.20.4/32 nexthop via 172.31.11.250 dev eth0 nexthop via 172.31.12.250 dev eth1 ``` and like this to the loopback address of every other node in the same rack: ```text 172.31.10.4/32 nexthop dev eth0 nexthop dev eth1 ``` If you launch some pods in the cluster, you should see ECMP routes for the /26 IP blocks for the nodes where those pods were scheduled, like this: ```text 10.244.192.128/26 nexthop via 172.31.11.250 dev eth0 nexthop via 172.31.12.250 dev eth1 ``` If you do something to break the connectivity between racks of one of the planes, you should see, within only a few seconds, that the affected routes change to have a single path only, via the plane that is still unbroken; for example: ```text 172.31.20.4/32 via 172.31.12.250 dev eth1` 10.244.192.128/26 via 172.31.12.250 dev eth1 ``` When the connectivity break is repaired, those routes should change to become ECMP again. ### Configure multiple Calico Cloud networks on a pod ## Big picture[​](#big-picture) Configure a Kubernetes cluster with multiple Calico Cloud networks on each pod, and enforce security using Calico Cloud tiered network policy. ## Value[​](#value) By default, you can configure only one CNI (network and pod interface) in a cluster. But many deployments require multiple networks (for example, one that is faster or more secure) for sending different types of data. Calico Cloud supports configuring additional Calico Cloud networks and interfaces in your pods using the Multus-CNI plugin. You can then use Calico Cloud tiered policy and other features to enforce security on all of your workload traffic. ## Concepts[​](#concepts) ### About the Multus-CNI plugin[​](#about-the-multus-cni-plugin) Calico Cloud uses the [Multus-CNI plugin](https://github.com/intel/multus-cni/) to create multiple Calico Cloud networks and multiple pod interfaces to access these networks. This extends the default network and pod interface that comes with the Calico CNI. You install Multus on a cluster, then simply enable Multus in the Calico Cloud Installation resource. Using the Multus **NetworkAttachmentDefinition**, you define the new networks and reference them as an annotation in the pod resource. ### Labels, workload endpoints, and policy[​](#labels-workload-endpoints-and-policy) When you set the `MultiInterfaceMode` field to `Multus` in the Installation resource, the following network and network interface labels are automatically added to new workload endpoints. - `projectcalico.org/network` - `projectcalico.org/network-namespace` - `projectcalico.org/network-interface` You can then create Calico Cloud policies using these label selectors to target specific networks or network interfaces. ### Limitations[​](#limitations) **Maximum additional networks per pod** You can define a maximum of nine additional Calico Cloud networks on a pod. If you add a network that exceeds the limit for the pod, networking is not configured and the pod fails to start with an associated error. **Calico Cloud features** Although the following Calico Cloud features are supported for your default Calico Cloud network, they are not supported at this time for additional networks/network interfaces using Multus: - Floating IPs - Specific IPs - Specifying IP pools on a per-namespace or per-pod basis - Egress gateways ## Before you begin...[​](#before-you-begin) **Required** - Calico CNI > **SECONDARY:** Verify that you are using the Calico Cloud CNI. The CNI plugin used by Kubernetes for AKS, EKS, and GKE may be different, which means this feature will not work. - [Install Multus 3.0+ on your Kubernetes cluster](https://github.com/intel/multus-cni/) > **SECONDARY:** Multus is installed on OpenShift 4.0+ clusters. ## How to[​](#how-to) 1. [Configure cluster for multiple networks](#configure-cluster-for-multiple-networks) 2. [Create a new network](#create-a-new-network) 3. [Create a pod interface for the new network](#create-a-pod-interface-for-the-new-network) 4. [Configure the IP pool for the network](#configure-the-ip-pool-for-the-network) 5. [Enforce policy on the new network and pod interface](#enforce-policy-on-the-new-network-and-pod-interface) 6. [View workload endpoints](#view-workload-endpoints) ### Configure cluster for multiple networks[​](#configure-cluster-for-multiple-networks) In the [Installation custom resource](https://docs.tigera.io/calico-cloud/reference/installation/api#caliconetworkspec), set the `MultiInterfaceMode` to **Multus**. ### Create a new network[​](#create-a-new-network) Create a new network using the Multus **NetworkAttachmentDefinition**, and set the following required field to `"type":"calico"`. ```yaml apiVersion: 'k8s.cni.cncf.io/v1' kind: NetworkAttachmentDefinition metadata: name: additional-calico-network spec: config: '{ "cniVersion": "0.3.1", "type": "calico", "log_level": "info", "datastore_type": "kubernetes", "mtu": 1410, "nodename_file_optional": false, "ipam": { "type": "calico-ipam", "assign_ipv4" : "true", "assign_ipv6" : "false" }, "policy": { "type": "k8s" }, "kubernetes": { "kubeconfig": "/etc/cni/net.d/calico-kubeconfig" } }' ``` ### Create a pod interface for the new network[​](#create-a-pod-interface-for-the-new-network) Create a pod interface that specifies the new network using an annotation. In the following example, we create a pod with an additional pod interface named, `cali1`. The pod interface is attached to the network named, `additional-calico-network`, using the `k8s.v1.cni.cncf.io/networks` annotation. Note that all networks in `k8s.v1.cni.cncf.io/networks` are assumed to be Calico Cloud networks. ```yaml apiVersion: v1 kind: Pod metadata: name: multus-test-pod-1 namespace: default annotations: k8s.v1.cni.cncf.io/networks: additional-calico-network@cali1 spec: nodeSelector: kubernetes.io/os: linux containers: - name: multus-test command: ['/bin/sh', '-c', 'trap : TERM INT; sleep infinity & wait'] image: alpine ``` ### Configure the IP pool for the network[​](#configure-the-ip-pool-for-the-network) Although not required, you may want to assign IPs from specific pools to specific network interfaces. If you are using the [Calico Cloud IPAM plugin](https://docs.tigera.io/calico-cloud/reference/component-resources/configuration#specifying-ip-pools-on-a-per-namespace-or-per-pod-basis), specify the IP pools in the **NetworkAttachmentDefinition** custom resource. For example: ```text "ipam": { "type": "calico-ipam", "assign_ipv4" : "true", "assign_ipv6" : "false" "ipv4_pools": ["10.0.0.0/24", "20.0.0.0/16", "default-ipv4-ippool"], }, ``` ### Enforce policy on the new network and pod interface[​](#enforce-policy-on-the-new-network-and-pod-interface) When MultiInterfaceMode is set to Multus, WorkloadEndpoints are created with these labels: - `projectcalico.org/network` - `projectcalico.org/network-namespace` - `projectcalico.org/network-interface` You can use these labels to enforce policies on specific interfaces and networks using policy label selectors. > **SECONDARY:** Prior to Calico Cloud 3.0, if you were using Kubernetes datastore (kdd mode), the workload endpoint field and name suffix were always **eth0**. In 3.0, the value for workload labels may not be what you expect. Before creating policies targeting WorkloadEndpoints using the new labels, you should verify label values using the commands in [View workload endpoints](#view-workload-endpoints). In this policy example, we use the selector field to target all WorkloadEndpoints with the network interface of, `cali1`. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: internal-access.allow-tcp-6379 namespace: production spec: tier: internal-access selector: projectcalico.org/network-interface == cali1 types: - Ingress - Egress ingress: - action: Allow metadata: annotations: from: frontend to: database protocol: TCP source: selector: role == 'frontend' destination: ports: - 6379 egress: - action: Allow ``` ### View workload endpoints[​](#view-workload-endpoints) **In the Calico Cloud web console**, go to the **WorkloadEndpoint** page to see all of the WorkloadEndpoints, including the network labels are for targeting WorkloadEndpoints with policy. **Using the CLI...** To view all WorkloadEndpoints for pods (default and new), use the following command. ```text MULTI_INTERFACE_MODE=multus calicoctl get workloadendpoints -o wide ``` ```text NAME WORKLOAD NODE NETWORKS INTERFACE PROFILES NATS test--bo--72vg--kadm--infra--0-k8s-multus--test--pod--1-eth0 multus-test-pod-1 bryan-bo-72vg-kadm-infra-0 192.168.53.129/32 calif887e436e8b kns.default,ksa.default.default test--bo--72vg--kadm--infra--0-k8s-multus--test--pod--1-net1 multus-test-pod-1 bryan-bo-72vg-kadm-infra-0 192.168.53.140/32 calim17CD6INXIX kns.default,ksa.default.default test--bo--72vg--kadm--infra--0-k8s-multus--test--pod--1-testiface multus-test-pod-1 bryan-bo-72vg-kadm-infra-0 192.168.53.142/32 calim27CD6INXIX kns.default,ksa.default.default test--bo--72vg--kadm--infra--0-k8s-multus--test--pod--1-net3 multus-test-pod-1 bryan-bo-72vg-kadm-infra-0 192.168.52.143/32 calim37CD6INXIX kns.default,ksa.default.default ``` To view specific WorkloadEndpoints, use the following command. ```text MULTI_INTERFACE_MODE=multus calicoctl get workloadendpoint test--bz--72vg--kadm--infra--0-k8s-multus--test--pod--1-net1 -o yaml ``` ```yaml apiVersion: projectcalico.org/v3 kind: WorkloadEndpoint metadata: creationTimestamp: '2020-05-04T22:23:05T' labels: projectcalico.org/namespace: default projectcalico.org/network: calico projectcalico.org/network-interface: net1 projectcalico.org/network-namespace: default projectcalico.org/orchestrator: k8s projectcalico.org/serviceaccount: default name: test--bz--72vg--kadm--infra--0-k8s-multus--test--pod--1-net1 namespace: default resourceVersion: '73572' uid: b9bb7482-cdb8-48d4-9ae5-58322d48391a spec: endpoint: net1 interfaceName: calim16CD6INXIX ipNetworks: - 192.168.52.141/32 node: bryan-bo-72vg-kadm-infra-0 orchestrator: k8s pod: multus-test-pod-1 profiles: - kns.default - ksa.default.default ``` ### Overlay networking ## Big picture[​](#big-picture) Enable inter workload communication across networks that are not aware of workload IPs. ## Value[​](#value) In general, we recommend running Calico without network overlay/encapsulation. This gives you the highest performance and simplest network; the packet that leaves your workload is the packet that goes on the wire. However, selectively using overlays/encapsulation can be useful when running on top of an underlying network that cannot easily be made aware of workload IPs. A common example is if you are using Calico networking in AWS across multiple VPCs/subnets. In this case, Calico can selectively encapsulate only the traffic that is routed between the VPCs/subnets, and run without encapsulation within each VPC/subnet. You might also decide to run your entire Calico network with encapsulation as an overlay network -- as a quick way to get started without setting up BGP peering or other routing information in your underlying network. ## Concepts[​](#concepts) ### Routing workload IP addresses[​](#routing-workload-ip-addresses) Networks become aware of workload IP addresses through layer 3 routing techniques like static routes or BGP route distribution, or layer 2 address learning. As such, they can route unencapsulated traffic to the right host for the endpoint that is the ultimate destination. However, not all networks are able to route workload IP addresses. For example, public cloud environments where you don’t own the hardware, AWS across VPC subnet boundaries, and other scenarios where you cannot peer Calico over BGP to the underlay, or easily configure static routes. This is why Calico supports encapsulation, so you can send traffic between workloads without requiring the underlying network to be aware of workload IP addresses. ### Encapsulation types[​](#encapsulation-types) Calico supports two types of encapsulation: VXLAN and IP in IP. VXLAN is supported in some environments where IP in IP is not (for example, Azure). VXLAN has a slightly higher per-packet overhead because the header is larger, but unless you are running very network intensive workloads the difference is not something you would typically notice. The other small difference between the two types of encapsulation is that Calico's VXLAN implementation does not use BGP, whereas Calico's IP in IP implementation uses BGP between Calico nodes. ### Cross-subnet[​](#cross-subnet) Encapsulation of workload traffic is typically required only when traffic crosses a router that is unable to route workload IP addresses on its own. Calico can perform encapsulation on: all traffic, no traffic, or only on traffic that crosses a subnet boundary. ## Before you begin[​](#before-you-begin) **Required** - Calico CNI **Not supported** - Calico for OpenStack (i.e. when Calico is used as the Neutron plugin) **Limitations** - IP in IP supports only IPv4 addresses - VXLAN in IPv6 is only supported for kernel versions ≥ 4.19.1 or redhat kernel version ≥ 4.18.0 ## How to[​](#how-to) - [Configure default IP pools at install time](#configure-default-ip-pools-at-install-time) - [Configure IP in IP encapsulation for only cross-subnet traffic](#configure-ip-in-ip-encapsulation-for-only-cross-subnet-traffic) - [Configure IP in IP encapsulation for all inter workload traffic](#configure-ip-in-ip-encapsulation-for-all-inter-workload-traffic) - [Configure VXLAN encapsulation for only cross-subnet traffic](#configure-vxlan-encapsulation-for-only-cross-subnet-traffic) - [Configure VXLAN encapsulation for all inter workload traffic](#configure-vxlan-encapsulation-for-all-inter-workload-traffic) ### IPv4/6 address support[​](#ipv46-address-support) IP in IP supports only IPv4 addresses. ### Best practice[​](#best-practice) Calico has an option to selectively encapsulate only traffic that crosses subnet boundaries. We recommend using the **cross-subnet** option with IP in IP or VXLAN to minimize encapsulation overhead. Cross-subnet mode provides better performance in AWS multi-AZ deployments, Azure VNETs, and on networks where routers are used to connect pools of nodes with L2 connectivity. Be aware that switching encapsulation modes can cause disruption to in-progress connections. Plan accordingly. ### Configure default IP pools at install time[​](#configure-default-ip-pools-at-install-time) Default IP pools are configured at install-time automatically by Calico. For operator managed clusters, you can configure encapsulation in the IP pools section of the default Installation. For example, the following installation snippet will enable VXLAN across subnets. ```yaml kind: Installation apiVersion: operator.tigera.io/v1 metadata: name: default spec: calicoNetwork: ipPools: - cidr: 192.168.0.0/16 encapsulation: VXLANCrossSubnet ``` ### Configure IP in IP encapsulation for only cross-subnet traffic[​](#configure-ip-in-ip-encapsulation-for-only-cross-subnet-traffic) IP in IP encapsulation can be performed selectively, and only for traffic crossing subnet boundaries. To enable this feature, set `ipipMode` to `CrossSubnet`. ```yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: ippool-ipip-cross-subnet-1 spec: cidr: 192.168.0.0/16 ipipMode: CrossSubnet natOutgoing: true ``` ### Configure IP in IP encapsulation for all inter workload traffic[​](#configure-ip-in-ip-encapsulation-for-all-inter-workload-traffic) With `ipipMode` set to `Always`, Calico routes traffic using IP in IP for all traffic originating from a Calico enabled-host, to all Calico networked containers and VMs within the IP pool. ```yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: ippool-ipip-1 spec: cidr: 192.168.0.0/16 ipipMode: Always natOutgoing: true ``` ### Configure VXLAN encapsulation for only cross subnet traffic[​](#configure-vxlan-encapsulation-for-only-cross-subnet-traffic) VXLAN encapsulation can be performed selectively, and only for traffic crossing subnet boundaries. To enable this feature, set `vxlanMode` to `CrossSubnet`. ```yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: ippool-vxlan-cross-subnet-1 spec: cidr: 192.168.0.0/16 vxlanMode: CrossSubnet natOutgoing: true ``` ### Configure VXLAN encapsulation for all inter workload traffic[​](#configure-vxlan-encapsulation-for-all-inter-workload-traffic) With `vxlanMode` set to `Always`, Calico routes traffic using VXLAN for all traffic originating from a Calico enabled host, to all Calico networked containers and VMs within the IP pool. ```yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: ippool-vxlan-1 spec: cidr: 192.168.0.0/16 vxlanMode: Always natOutgoing: true ``` ## Additional resources[​](#additional-resources) For details on IP pool resource options, see [IP pool](https://docs.tigera.io/calico-cloud/reference/resources/ippool). ### Advertise Kubernetes service IP addresses ## Big picture[​](#big-picture) Enable Calico Cloud to advertise Kubernetes service IPs outside a cluster. Calico Cloud supports advertising a service’s cluster IPs and external IPs. ## Value[​](#value) Typically, Kubernetes service cluster IPs are accessible only within the cluster, so external access to the service requires a dedicated load balancer or ingress controller. In cases where a service’s cluster IP is not routable, the service can be accessed using its external IP. Just as Calico Cloud supports advertising **pod IPs** over BGP, it also supports advertising Kubernetes **service IPs** outside a cluster over BGP. This avoids the need for a dedicated load balancer. This feature also supports equal cost multi-path (ECMP) load balancing across nodes in the cluster, as well as source IP address preservation for local services when you need more control. ## Concepts[​](#concepts) ### BGP makes it easy[​](#bgp-makes-it-easy) In Kubernetes, all requests for a service are redirected to an appropriate endpoint (pod) backing that service. Because Calico Cloud uses BGP, external traffic can be routed directly to Kubernetes services by advertising Kubernetes service IPs into the BGP network. If your deployment is configured to peer with BGP routers outside the cluster, those routers (plus any other upstream places the routers propagate to) can send traffic to a Kubernetes service IP for routing to one of the available endpoints for that service. ### Advertising service IPs: quick glance[​](#advertising-service-ips-quick-glance) Calico Cloud implements the Kubernetes **externalTrafficPolicy** using kube-proxy to direct incoming traffic to a correct pod. Advertisement is handled differently based on the service type that you configure for your service. | **Service mode** | **Cluster IP advertisement** | **Traffic is...** | Source IP address is... | | ----------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------- | | Cluster (default) | All nodes in the cluster statically advertise a route to the service CIDR. | Load balanced across nodes in the cluster using ECMP, then forwarded to appropriate pod in the service using SNAT. May incur second hop to another node, but good overall load balancing. | Obscured by SNAT | | Local | The nodes with a pod backing the service advertise a specific route (/32 or /128) to the service's IP. | Load balanced across nodes with endpoints for the service. Avoids second hop for LoadBalancer and NodePort type services, traffic may be unevenly load balanced. (Other traffic is load balanced across nodes in the cluster.) | Preserved | If your Calico Cloud deployment is configured to peer with BGP routers outside the cluster, those routers - plus any further upstream places that those routers propagate to - will be able to send traffic to a Kubernetes service cluster IP, and that traffic is routed to one of the available endpoints for that service. ### Tips for success[​](#tips-for-success) - Generally, we recommend using “Local” for the following reasons: - If any of your network policy uses rules to match by specific source IP addresses, using Local is the obvious choice because the source IP address is not altered, and the policy will still work. - Return traffic is routed directly to the source IP because “Local” services do not require undoing the source NAT (unlike “Cluster” services). - Cluster IP advertisement works best with a ToR that supports ECMP. Otherwise, all traffic for a given route is directed to a single node. ## Before you begin...[​](#before-you-begin) **Required** - Calico CNI - [Configure BGP peering](https://docs.tigera.io/calico-cloud/networking/configuring/bgp) between Calico Cloud and your network infrastructure - For ECMP load balancing to services, the upstream routers must be configured to use BGP multipath. - You need at least one external node outside the cluster that acts as a router, route reflector, or ToR that is peered with calico nodes inside the cluster. - Services must be configured with the correct service mode (“Cluster” or “Local”) for your implementation. For `externalTrafficPolicy: Local`, the service must be type `LoadBalancer` or `NodePort`. **Limitations** - Supported in EKS and AWS, but only if you are using Calico CNI - OpenShift, versions 4.5 and 4.6\ There is a [bug](https://github.com/kubernetes/kubernetes/issues/91374) where the source IP is not preserved by NodePort services or traffic via a Service ExternalIP with externalTrafficPolicy:Local. OpenShift users on v4.5 or v4.6 can use this [workaround to avoid SNAT with ExternalIP](https://docs.openshift.com/container-platform/4.7/nodes/clusters/nodes-cluster-enabling-features.html): ```text oc edit featuregates.config.openshift.io cluster spec: customNoUpgrade: enabled: - ExternalPolicyForExternalIP ``` Kubernetes users on version v1.18 or v1.19 can enable source IP preservation for NodePort services using the ExternalPolicyForExternalIP feature gate. Source IP preservation for NodePort and services and ExternalIPs is enabled by default in OpenShift v4.7+, and Kubernetes v1.20+. ## How to[​](#how-to) - [Advertise service cluster IP addresses](#advertise-service-cluster-ip-addresses) - [Advertise service external IP addresses](#advertise-service-external-ip-addresses) - [Advertise service load balancer IP addresses](#advertise-service-load-balancer-ip-addresses) - [Exclude certain nodes from advertisement](#exclude-certain-nodes-from-advertisement) ### Advertise service cluster IP addresses[​](#advertise-service-cluster-ip-addresses) 1. Determine the service cluster IP range. (Or ranges, if your cluster is [dual stack](https://docs.tigera.io/calico-cloud/networking/ipam/ipv6).) The range(s) for your cluster can be inferred from the `--service-cluster-ip-range` option passed to the Kubernetes API server. For help, see the [Kubernetes API server reference guide](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/). 2. Check to see if you have a default BGPConfiguration. ```bash kubectl get bgpconfiguration.projectcalico.org default ``` 3. Based on above results, update or create a BGPConfiguration. **Update default BGPConfiguration**. Patch the BGPConfiguration using the following command, using your own service cluster IP CIDR in place of "10.0.0.0/24": ```bash kubectl patch bgpconfiguration.projectcalico.org default -p '{"spec":{"serviceClusterIPs": [{"cidr": "10.0.0.0/24"}]}}' ``` **Create default BGPConfiguration**. Use the following sample command to create a default BGPConfiguration. Add your CIDR blocks, covering the cluster IPs to be advertised, in the `serviceClusterIPs` field, for example: ```bash kubectl create -f - < **SECONDARY:** In earlier versions of Calico Cloud, and for IPv4 only, service cluster IP advertisement was configured via the environment variable CALICO\_ADVERTISE\_CLUSTER\_IPS. That environment variable takes precedence over any serviceClusterIPs configured in the default BGPConfiguration. We recommend replacing the deprecated CALICO\_ADVERTISE\_CLUSTER\_IPS with BGPConfiguration. #### Advertise an exact route for a particular service[​](#advertise-an-exact-route-for-a-particular-service) Service cluster IP advertisement of /32 routes, by default, is enabled only for `NodePort` and `LoadBalancer` type services. In some routing infrastructures, you may want to advertise /32 routes of `ClusterIP` services as well. This is enabled on a per-service basis by adding the annotation `projectcalico.org/AdvertiseClusterIP: true` to the service. For example: ```bash kubectl annotate service your-service "projectcalico.org/AdvertiseClusterIP=true" ``` > **SECONDARY:** You will still need to enable service cluster IP advertisement via BGP configuration as shown above. ### Advertise service external IP addresses[​](#advertise-service-external-ip-addresses) 1. Identify the external IP ranges of all services that you want to advertise outside of the Calico Cloud cluster. 2. Check to see if you have a default BGPConfiguration. ```bash kubectl get bgpconfiguration.projectcalico.org default ``` 3. Based on above results, update or create a BGPConfiguration. **Update default BGPConfiguration** Patch the BGPConfiguration using the following command, adding your own service external IP CIDRs: ```bash kubectl patch bgpconfiguration.projectcalico.org default -p '{"spec":{"serviceExternalIPs": [{"cidr": "x.x.x.x"}, {"cidr": "y.y.y.y"}]}}' ``` **Create default BGPConfiguration** Use the following sample command to create a default BGPConfiguration. Add your CIDR blocks for external IPs to be advertised in the `serviceExternalIPs` field. ```bash kubectl create -f - < **SECONDARY:** If you have previously used alternative IPAM solution for Service of type LoadBalancer address management and wish to continue using it, do not create Calico IPPool for LoadBalancer. ### Exclude certain nodes from advertisement[​](#exclude-certain-nodes-from-advertisement) In some cases, you may want to exclude certain nodes from advertising service addresses. For example, control plane nodes that do not host any services themselves. To remove a node from service advertisement, apply the label `node.kubernetes.io/exclude-from-external-load-balancers=true`. For example, to exclude the node `control-plane-01` from service advertisement, you can run the following command: ```text kubectl label node control-plane-01 node.kubernetes.io/exclude-from-external-load-balancers=true ``` ## Tutorial[​](#tutorial) For a tutorial on how service advertisement works with Calico Cloud, see the blog [Kubernetes Service IP Route Advertisement](https://www.projectcalico.org/kubernetes-service-ip-route-advertisement/). ## Additional resources[​](#additional-resources) Other topics on creating network policy for Kubernetes services: - [Apply policy to services exposed externally as cluster IPs](https://docs.tigera.io/calico-cloud/network-policy/beginners/services/services-cluster-ips) - [Apply policy to Kubernetes node ports](https://docs.tigera.io/calico-cloud/network-policy/beginners/services/kubernetes-node-ports) ### Configure MTU to maximize network performance ## Big picture[​](#big-picture) Configure the maximum transmission unit (MTU) for your Calico Cloud environment. ## Value[​](#value) Optimize network performance for workloads by configuring the MTU in Calico Cloud to best suit your underlying network. Increasing the MTU can improve performance, and decreasing the MTU can resolve packet loss and fragmentation problems when it is too high. ## Concepts[​](#concepts) ### MTU and Calico Cloud defaults[​](#mtu-and-calico-cloud-defaults) The maximum transmission unit (MTU) setting determines the largest packet size that can be transmitted through your network. MTU is configured on the veth attached to each workload, and tunnel devices (if you enable IP in IP, VXLAN, or WireGuard). In general, maximum performance is achieved by using the highest MTU value that does not cause fragmentation or dropped packets on the path. Maximum bandwidth increases and CPU consumption may drop for a given traffic rate. The improvement is often more significant when pod to pod traffic is being encapsulated (IP in IP, VXLAN, or WireGuard), and splitting and combining such traffic cannot be offloaded to your NICs. By default, Calico Cloud will auto-detect the correct MTU for your cluster based on node configuration and enabled networking modes. This guide explains how you can override auto-detection of MTU by providing an explicit value if needed. To ensure auto-detection of MTU works correctly, make sure that the correct encapsulation modes are set in your [felix configuration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig). Disable any unused encapsulations (`vxlanEnabled`, `ipipEnabled`, `wireguardEnabled` and `wireguardEnabledV6`) in your felix configuration to ensure that auto-detection can pick the optimal MTU for your cluster. ## Before you begin...[​](#before-you-begin) **Required** - Calico CNI For help on using IP in IP and/or VXLAN overlays, see [Configure overlay networking](https://docs.tigera.io/calico-cloud/networking/configuring/vxlan-ipip). For help on using WireGuard encryption, see [Configure WireGuard encryption](https://docs.tigera.io/calico-cloud/compliance/encrypt-cluster-pod-traffic). ## How to[​](#how-to) - [Determine MTU size](#determine-mtu-size) - [Configure MTU](#configure-mtu) - [View current tunnel MTU values](#view-current-tunnel-mtu-values) ### Determine MTU size[​](#determine-mtu-size) The following table lists common MTU sizes for Calico Cloud environments. Because MTU is a global property of the network path between endpoints, you should set the MTU to the minimum MTU of any path that packets may take. **Common MTU sizes** | Network MTU | Calico Cloud MTU | Calico Cloud MTU with IP-in-IP (IPv4) | Calico Cloud MTU with VXLAN (IPv4) | Calico Cloud MTU with VXLAN (IPv6) | Calico Cloud MTU with WireGuard (IPv4) | Calico Cloud MTU with WireGuard (IPv6) | | ---------------------- | ---------------- | ------------------------------------- | ---------------------------------- | ---------------------------------- | -------------------------------------- | -------------------------------------- | | 1500 | 1500 | 1480 | 1450 | 1430 | 1440 | 1420 | | 9000 | 9000 | 8980 | 8950 | 8930 | 8940 | 8920 | | 1500 (AKS) | 1500 | 1480 | 1450 | 1430 | 1340 | 1320 | | 1460 (GCE) | 1460 | 1440 | 1410 | 1390 | 1400 | 1380 | | 9001 (AWS Jumbo) | 9001 | 8981 | 8951 | 8931 | 8941 | 8921 | | 1450 (OpenStack VXLAN) | 1450 | 1430 | 1400 | 1380 | 1390 | 1370 | **Recommended MTU for overlay networking** The extra overlay header used in IP in IP, VXLAN and WireGuard protocols, reduces the minimum MTU by the size of the header. (IP in IP uses a 20-byte header, IPv4 VXLAN uses a 50-byte header, IPv6 VXLAN uses a 70-byte header, IPv4 WireGuard uses a [60-byte header](https://lists.zx2c4.com/pipermail/wireguard/2017-December/002201.html) and IPv6 WireGuard uses an 80-byte header). When using AKS, the underlying network has an [MTU of 1400](https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-tcpip-performance-tuning#azure-and-vm-mtu), even though the network interface will have an MTU of 1500. WireGuard sets the Don't Fragment (DF) bit on its packets, and so the MTU for WireGuard on AKS needs to be set to 60 bytes below (or 80 bytes for IPv6) the 1400 MTU of the underlying network to avoid dropped packets. If you have a mix of WireGuard and either IP in IP or VXLAN in your cluster, you should configure the MTU to be the smallest of the values of each encap type. The reason for this is that only WireGuard encapsulation will be used between any nodes where both have WireGuard enabled, and IP in IP or VXLAN will then be used between any nodes where both do not have WireGuard enabled. This could be the case if, for example, you are in the process of installing WireGuard on your nodes. Therefore, we recommend the following: - If you use IPv4 WireGuard encryption anywhere in your pod network, configure MTU size as “physical network MTU size minus 60”. - If you use IPv6 WireGuard encryption anywhere in your pod network, configure MTU size as “physical network MTU size minus 80”. - If you don't use WireGuard, but use IPv4 VXLAN anywhere in your pod network, configure MTU size as “physical network MTU size minus 50”. - If you don't use WireGuard, but use IPv6 VXLAN anywhere in your pod network, configure MTU size as “physical network MTU size minus 70”. - If you don't use WireGuard, but use only IP in IP, configure MTU size as “physical network MTU size minus 20” - Set the workload endpoint MTU and the tunnel MTUs to the same value (so all paths have the same MTU) **eBPF mode** Implementation of NodePorts uses VXLAN tunnel to hand off packets from one node to another, therefore VXLAN MTU setting is used to set the MTUs of workloads (veths) and should be “physical network MTU size minus 50” (see above). **MTU for flannel networking** When using flannel for networking, the MTU for network interfaces should match the MTU of the flannel interface. - If using flannel with VXLAN, use the “Calico Cloud MTU with VXLAN” column in the table above for common sizes. ### Configure MTU[​](#configure-mtu) > **SECONDARY:** The updated MTU used by Calico Cloud only applies to new workloads. For Operator installations, edit the Calico Cloud operator `Installation` resource to set the `mtu` field in the `calicoNetwork` section of the `spec`. For example: ```bash kubectl patch installation.operator.tigera.io default --type merge -p '{"spec":{"calicoNetwork":{"mtu":1440}}}' ``` Similarly, for OpenShift: ```bash oc patch installation.operator.tigera.io default --type merge -p '{"spec":{"calicoNetwork":{"mtu":1440}}}' ``` ### View current tunnel MTU values[​](#view-current-tunnel-mtu-values) To view the current tunnel size, use the following command: `ip link show` The IP in IP tunnel appears as tunlx (for example, tunl0), along with the MTU size. For example: ![Tunnel MTU](https://docs.tigera.io/assets/images/tunnel-ff211553ee61a6bc8150ae139fa154ba.png) ### Custom BGP Configuration ## Big picture[​](#big-picture) Use customized BIRD configuration files to enable specialized use-cases. ## Concepts[​](#concepts) In Calico Cloud, BGP is handled by [BIRD](https://github.com/projectcalico/bird). The BIRD configurations are templated out through [confd](https://github.com/projectcalico/confd). You can modify the BIRD configuration to use BIRD features which are not typically exposed using the default configuration provided with Calico Cloud. Customization of BGP templates should be done only with the help of your Tigera Support representative. ## Before you begin[​](#before-you-begin) **Required** - Calico CNI ## How to[​](#how-to) - [Update BGP configuration](#update-bgp-configuration) - [Apply BGP customizations](#apply-bgp-customizations) based on how you've deployed Calico Cloud: ### Update BGP configuration[​](#update-bgp-configuration) Using the directions provided with the templates, set the correct values for the BGP configuration using these resources: - [BGP Configuration](https://docs.tigera.io/calico-cloud/reference/resources/bgpconfig) - [BGP Peer](https://docs.tigera.io/calico-cloud/reference/resources/bgppeer) ### Apply BGP Customizations[​](#apply-bgp-customizations) 1. Create your confd templates. 2. Create a ConfigMap from the templates. ```text kubectl create configmap bird-templates -n tigera-operator --from-file= ``` The created config map will be used to populate the Calico Cloud BIRD configuration file templates. If a template with the same name already exists within the node container, it will be overwritten with the contents from the config map. ### Configure outgoing NAT ## Big picture[​](#big-picture) Configure Calico Cloud networking to perform outbound NAT for connections from pods to outside of the cluster. Calico Cloud optionally source NATs the pod IP to the node IP. ## Value[​](#value) The Calico Cloud NAT outbound connection option is flexible; it can be enabled, disabled, and applied to Calico Cloud IP pools with public IPs, private IPs, or a specific range of IP addresses. This article describes some use cases for enabling and disabling outgoing NAT. ## Concepts[​](#concepts) ### Calico Cloud IP pools and NAT[​](#calico-cloud-ip-pools-and-nat) When a pod with an IP address in the pool initiates a network connection to an IP address to outside of Calico Cloud’s IP pools, the outgoing packets will have their source IP address changed from the pod IP address to the node IP address using SNAT (Source Network Address Translation). Any return packets on the connection automatically get this change reversed before being passed back to the pod. ### Enable NAT: for pods with IP addresses that are not routable beyond the cluster[​](#enable-nat-for-pods-with-ip-addresses-that-are-not-routable-beyond-the-cluster) A common use case for enabling NAT outgoing, is to allow pods in an overlay network to connect to IP addresses outside of the overlay, or pods with private IP addresses to connect to public IP addresses outside the cluster/the internet (subject to network policy allowing the connection, of course). When NAT is enabled, traffic is NATed from pods in that pool to any destination outside of all other Calico Cloud IP pools. ### Disable NAT: For on-premises deployments using physical infrastructure[​](#disable-nat-for-on-premises-deployments-using-physical-infrastructure) If you choose to implement Calico Cloud networking with [BGP peered with your physical network infrastructure](https://docs.tigera.io/calico-cloud/networking/configuring/bgp), you can use your own infrastructure to NAT traffic from pods to the internet. In this case, you should disable the Calico Cloud `natOutgoing` option. For example, if you want your pods to have public internet IPs, you should: - Configure Calico Cloud to peer with your physical network infrastructure - Create an IP pool with public IP addresses for those pods that are routed to your network with NAT disabled (`natOutgoing: false`) - Verify that other network equipment does not NAT the pod traffic ## Before you begin[​](#before-you-begin) **Required** - Calico CNI ## How to[​](#how-to) - [Create an IP pool with NAT outgoing enabled](#create-an-ip-pool-with-nat-outgoing-enabled) - [Use additional IP pools to specify addresses that can be reached without NAT](#use-additional-ip-pools-to-specify-addresses-that-can-be-reached-without-nat) ### Create an IP pool with NAT outgoing enabled[​](#create-an-ip-pool-with-nat-outgoing-enabled) In the following example, we create a Calico Cloud IPPool with natOutgoing enabled. Outbound NAT is performed locally on the node where each workload in the pool is hosted. ```yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: default-ipv4-ippool spec: cidr: 192.168.0.0/16 natOutgoing: true ``` ### Use additional IP pools to specify addresses that can be reached without NAT[​](#use-additional-ip-pools-to-specify-addresses-that-can-be-reached-without-nat) Because Calico Cloud performs outgoing NAT only when connecting to an IP address that is not in a Calico Cloud IPPool, you can create additional IPPools that are not used for pod IP addresses, but prevent NAT to certain CIDR blocks. This is useful if you want nodes to NAT traffic to the internet, but not to IPs in certain internal ranges. For example, if you did not want to NAT traffic from pods to 10.0.0.0/8, you could create the following pool. You must ensure that the network between the cluster and 10.0.0.0/8 can route pod IPs. ```yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: no-nat-10.0.0.0-8 spec: cidr: 10.0.0.0/8 disabled: true ``` ### Use a specific MAC address for a pod ## Big picture[​](#big-picture) Choose the MAC address for a pod instead of allowing the operating system to assign one. ## Value[​](#value) Some applications bind software licenses to networking interface MAC addresses. ## Concepts[​](#concepts) ### Container MAC address[​](#container-mac-address) The MAC address configured by the annotation described here will be visible from within the container on the eth0 interface. Since it is isolated to the container it will not collide with any other MAC addresses assigned to other pods on the same node. ## Before you begin...[​](#before-you-begin) Your cluster must be using Calico CNI to use this feature. [Configuring the Calico CNI Plugins](https://docs.tigera.io/calico-cloud/reference/component-resources/configuration) ## How to[​](#how-to) Annotate the pod with cni.projectcalico.org/hwAddr set to the desired MAC address. For example: ```text "cni.projectcalico.org/hwAddr": "1c:0c:0a:c0:ff:ee" ``` The annotation must be present when the pod is created; adding it later has no effect. ### Use NodeLocal DNSCache in your cluster ## Big picture[​](#big-picture) Set up NodeLocal DNSCache to improve DNS lookup latency. ## Before you begin[​](#before-you-begin) ### Required[​](#required) Follow these [steps](https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/) to enable NodeLocal DNSCache connectivity. ## Create a policy to allow traffic from NodeLocal DNSCache[​](#create-a-policy-to-allow-traffic-from-nodelocal-dnscache) The following is a sample network policy that allows all incoming TCP traffic (including incoming traffic from `node-local-dns` pods) on port 53 on `kube-dns`. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: default.local-dns-to-core-dns namespace: kube-system spec: tier: default selector: k8s-app == "kube-dns" ingress: - action: Allow protocol: TCP destination: selector: k8s-app == "kube-dns" ports: - '53' types: - Ingress ``` To refine the sources permitted by this policy, take into account that NodeLocal DNSCache pods are host networked, and make sure to allow traffic from the addresses of your hosts. If you're using encapsulation, you will need to allow connectivity from the tunnel IPs. The Tigera Operator creates policy to allow Tigera components to connect to NodeLocal DNSCache when detected. Felix accounts for the NodeLocal DNSCache in creating DNS Logs and enforcing DNS Policy. ## Enable NodeLocal DNSCache on Calico Cloud using the eBPF data plane[​](#enable-nodelocal-dnscache-on-calico-cloud-using-the-ebpf-data-plane) If your Calico Cloud installation uses the eBPF data plane, then you need to annotate the `kube-dns` service. Without this, the `kube-proxy` replacement will improperly resolve traffic going to a specific service IP. Annotate the `kube-dns` service by running the following command: ```bash kubectl annotate service kube-dns -n kube-system projectcalico.org/natExcludeService=true ``` ## Additional resources[​](#additional-resources) - [Blog: Optimizing Kubernetes DNS with NodeLocal DNSCache and Calico eBPF: A Practitioner’s Guide](https://www.tigera.io/blog/optimizing-kubernetes-dns-with-nodelocal-dnscache-and-calico-ebpf-a-practitioners-guide/) ### Configure QoS Controls ## Big picture[​](#big-picture) Configure QoS (Quality of Service) Controls to limit ingress and/or egress bandwidth, packet rate and number of connections used by Calico Cloud workloads in order to prevent them from overusing or overloading network resources. Additionally, it allows configuring Differentiated Services (DiffServ) on egress traffic of Calico Cloud workloads and hosts. ## Value[​](#value) With QoS controls, Calico Cloud enforces network resource limits (bandwidth, packet rate, connections) for Kubernetes pods and OpenStack VMs, ensuring fair resource allocation and preventing performance degradation for other workloads. Calico Cloud can also apply DiffServ on egress traffic to allow upstream devices to prioritise forwarding accordingly. ## Limitations[​](#limitations) - Configuring limits for established connections is not supported on installations that use the eBPF data plane. ## Concepts[​](#concepts) ### Quality of Service Controls[​](#quality-of-service-controls) **Quality of Service** (**QoS**), in the field of computer networking, refers to traffic prioritization and resource reservation control mechanisms. It may either mean guaranteeing minimum levels of networking resources, or limiting the maximum levels of usage of networking resources in order to prevent a small number of users from excessive resource consumption, stopping others from using their fair share. For practical purposes, the latter will be considered as Calico Cloud implements QoS Controls by limiting maximum levels of network resources usage. Calico Cloud implements three types of QoS Controls, which can be configured using annotations on the Kubernetes pods for either ingress (traffic coming into the pod), egress (traffic going out of the pod), or both: 1. Bandwidth: limiting the bit rate of traffic into/out of the pod 2. Packet rate: limiting the number of packets per second that can be sent or received by the pod 3. Number of established connections: limiting the absolute number of connections that the pod may initiate or accept 4. Differentiated Services (DiffServ): setting Differentiated Services Code Point (DSCP) on egress traffic leaving clusters or toward hosts ### Annotations[​](#annotations) On Kubernetes, Calico Cloud uses [annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) on pods for the configuration of QoS controls. The values of the QoS controls annotations may contain suffixes, such as `k`, `M`, `G`, `T`, `P`, or `E` to represent large numbers. Binary suffixes are also supported, for example `1Ki` represents "1024". For example, to limit egress bandwidth to 100Mbps, with a maximum burst of 200Mb: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-deployment labels: app: my-app spec: replicas: 3 template: metadata: annotations: qos.projectcalico.org/egressBandwidth: "100M" qos.projectcalico.org/egressBurst: "200M" spec: (...) ``` ## How to[​](#how-to) - [Configure bandwidth limits](#configure-bandwidth-limits) - [Configure packet rate limits](#configure-packet-rate-limits) - [Configure established connection limits](#configure-established-connection-limits) - [Configure DiffServ](#configure-diffserv) ### Configure bandwidth limits[​](#configure-bandwidth-limits) Use the annotations below to configure bandwidth and burst limits for traffic going into or coming out of the pod. Limiting the bandwidth enforces that, on average, that will be the maximum bit rate used. The burst size is the amount of data that is available instantaneously for transmission. Burst configuration is optional, if burst is not explicitly configured with an annotation, a default value of `4Gib` is used. If you configure a peak rate limit, burst traffic will not go over that limit. Otherwise, there are no limits to burst traffic. | Annotation | Description | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `qos.projectcalico.org/ingressBandwidth` | Specifies the ingress bandwidth limit (in bits per second). Min value: `1k` Max value: `1P` | | `qos.projectcalico.org/ingressBurst` | Specifies the ingress burst size (in bits). Optional, defaults to `4Gi`. Min value: `1k` Max value: `34359738360` | | `qos.projectcalico.org/ingressPeakrate` | Specifies the ingress peak rate limit, i.e. the bandwidth limit during bursts (in bits per second). Optional, defaults to unlimited. Must be higher than the bandwidth limit. Min value: `1.01k` Max value: `1.01P` | | `qos.projectcalico.org/ingressMinburst` | Specifies the ingress minburst, i.e. the burst size for peak rate (in **bytes**). Optional, defaults to the MTU value. Min value: `1k` Max value: `100M` | | `qos.projectcalico.org/egressBandwidth` | Specifies the egress bandwidth limit (in bits per second). Min value: `1k` Max value: `1P` | | `qos.projectcalico.org/egressBurst` | Specifies the egress burst size (in bits). Optional, defaults to `4Gi`. Min value: `1k` Max value: `34359738360` | | `qos.projectcalico.org/egressPeakrate` | Specifies the egress peak rate limit, i.e. the bandwidth limit during bursts (in bits per second). Optional, defaults to unlimited. Must be higher than the bandwidth limit. Min value: `1.01k` Max value: `1.01P` | | `qos.projectcalico.org/egressMinburst` | Specifies the egress minburst, i.e. the burst size for peak rate (in **bytes**). Optional, defaults to the MTU value. Min value: `1k` Max value: `100M` | If the [Kubernetes bandwidth plugin](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) annotations `kubernetes.io/ingress-bandwidth` or `kubernetes.io/egress-bandwidth` are configured and the `qos.projectcalico.org` annotations are not, they are honored and Calico Cloud will configure bandwidth limits based on them. The Calico Cloud specific annotations take precedence, so if both are configured, the Kubernetes annotation will be ignored. This makes upgrading from a Calico Cloud version that uses the bandwidth plugin to a version with native bandwith limiting seamless, as the behavior will remain consistent across versions (with Calico Cloud replacing the functionality previously provided by the Kubernetes bandwidth plugin). One essential difference between the bandwidth plugin and Calico Cloud's implementations is that with Calico Cloud, the bandwidth limits take effect immediately, whereas with the bandwidth plugin the pods need to be recreated for the limits to take effect. | Annotation | Description | | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `kubernetes.io/ingress-bandwidth` | Kubernetes bandwidth plugin annotation for the ingress bandwidth limit (in bits per second). Ignored if `qos.projectcalico.org/ingressBandwidth` is present. Min value: `1k` Max value: `1P` | | `kubernetes.io/egress-bandwidth` | Kubernetes bandwidth plugin annotation for the egress bandwidth limit (in bits per second). Ignored if `qos.projectcalico.org/egressBandwidth` is present. Min value: `1k` Max value: `1P` | ### Configure packet rate limits[​](#configure-packet-rate-limits) Use the annotations below to configure packet rate limits for traffic going into or coming out of the pod. Limiting the packet rate enforces that, on average, that will be the maximum rate that packets will be sent and/or received by the pod. The packet burst size is the number of packets that are available instantaneously for transmission. Packet burst configuration is optional. If packet burst is not explicitly configured with an annotation, a default value of `5` packets is used. | Annotation | Description | | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------- | | `qos.projectcalico.org/ingressPacketRate` | Specifies the maximum rate of ingress packets (in packets per second). Min value: `1` Max value: `10k` | | `qos.projectcalico.org/ingressPacketBurst` | Specifies the ingress packet burst size (in number of packets). Optional, defaults to `5`. Min value: `1` Max value: `10k` | | `qos.projectcalico.org/egressPacketRate` | Specifies the maximum rate of egress packets (in packets per second). Min value: `1` Max value: `10k` | | `qos.projectcalico.org/egressPacketBurst` | Specifies the egress packet burst size (in number of packets). Optional, defaults to `5`. Min value: `1` Max value: `10k` | ### Configure established connection limits[​](#configure-established-connection-limits) Use the annotations below to configure limits for the number of established connections into or out of a pod. Limiting the number of connections enforces that the pod cannot accept or initiate more connections that the specified limits. | Annotation | Description | | --------------------------------------------- | ------------------------------------------------------------------------------------------- | | `qos.projectcalico.org/ingressMaxConnections` | Specifies the maximum number of ingress connections. Min value: `1` Max value: `4294967295` | | `qos.projectcalico.org/egressMaxConnections` | Specifies the maximum number of egress connections. Min value: `1` Max value: `4294967295` | ### Configure DiffServ[​](#configure-diffserv) Use the annotation below to set DSCP on egress traffic toward an endpoint outside cluster or another host in cluster. Unlike, other QoS controls, this annotation can also be set on a [host endpoint](https://docs.tigera.io/calico-cloud/reference/host-endpoints). For example, to set DSCP AF11 on a host endpoint: ```yaml apiVersion: projectcalico.org/v3 kind: HostEndpoint metadata: name: my-hep annotations: qos.projectcalico.org/dscp: "AF11" labels: app: my-app spec: (...) ``` | Annotation | Description | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `qos.projectcalico.org/dscp` | Specifies the DSCP value of egress traffic toward an endpoint outside cluser or another host in cluster. Min value: `0` Max value: `63` String values: DF, EF, AF11, AF12, AF13, AF21, AF22, AF23, AF31, AF32, AF33, AF41, AF42, AF43, CS0, CS1, CS2, CS3, CS4, CS5, CS6 | ### Egress gateways ## [📄️ Configure egress gateways, on-premises](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-on-prem) [Configure specific application traffic to exit the cluster through an egress gateway.](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-on-prem) ## [📄️ Configure egress gateways, AWS](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-aws) [Configure specific application traffic to exit the cluster through an egress gateway with a native AWS IP address.](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-aws) ## [📄️ Configure egress gateways, Azure](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-azure) [Configure specific application traffic to exit the cluster through an egress gateway with a native Azure IP address.](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-azure) ## [📄️ Optimize egress networking for workloads with long-lived TCP connections](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-maintenance) [React to egress gateway maintenance windows and minimize the impact of egress gateway downtime on sensitive workloads](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-maintenance) ## [📄️ Configure egress traffic to multiple external networks](https://docs.tigera.io/calico-cloud/networking/egress/external-network) [Allows workloads from different namespaces of a Kubernetes cluster to egress onto different external networks that (may) have overlapping IPs with each other.](https://docs.tigera.io/calico-cloud/networking/egress/external-network) ## [📄️ Troubleshoot egress gateways](https://docs.tigera.io/calico-cloud/networking/egress/troubleshoot) [Use checklist to troubleshoot common problems.](https://docs.tigera.io/calico-cloud/networking/egress/troubleshoot) ### Configure egress gateways, on-premises ## Big picture[​](#big-picture) Configure specific application traffic to exit the cluster through an egress gateway. ## Value[​](#value) When traffic from particular applications leaves the cluster to access an external destination, it can be useful to control the source IP of that traffic. For example, there may be an additional firewall around the cluster, whose purpose includes policing external accesses from the cluster, and specifically that particular external destinations can only be accessed from authorised workloads within the cluster. Calico Cloud's own policy (including [DNS policy](https://docs.tigera.io/calico-cloud/network-policy/domain-based-policy)) and per-node firewalls can ensure this, but deployments may like to deepen their defense by adding an external firewall as well. If the external firewall is configured to allow outbound connections only from particular source IPs, and the intended cluster workloads can be configured so that their outbound traffic will have one of those source IPs, then the defense in depth objective is achieved. Calico Cloud allows specifying an [IP pool](https://docs.tigera.io/calico-cloud/reference/resources/ippool) for each pod or namespace, and even a [specific IP](https://docs.tigera.io/calico-cloud/networking/ipam/use-specific-ip) for a new pod, but this requires predicting how many pods there will be representing a particular application, so that the IP pool can be correctly sized. When IPs are a precious resource, over-sizing the pool is wasteful; but under-sizing is also problematic, as then IPs will not be available within the desired range as the application is scaled. Egress gateways provide an alternative approach. Application pods and namespaces are provisioned with IPs from the default (and presumably plentiful) pool, but also configured so that their outbound traffic is directed through an egress gateway. (Or, for resilience, through one of a small number of egress gateways.) The egress gateways are set up to use a [specific IP pool](https://docs.tigera.io/calico-cloud/networking/ipam/legacy-firewalls) and to perform an SNAT on the traffic passing through them. Hence, any number of application pods can have their outbound connections multiplexed through a fixed small number of egress gateways, and all of those outbound connections acquire a source IP from the egress gateway IP pool. > **SECONDARY:** The source port of an outbound flow through an egress gateway can generally *not* be preserved. Changing the source port is how Linux maps flows from many upstream IPs onto a single downstream IP. Egress gateways are also useful if there is a reason for wanting all outbound traffic from a particular application to leave the cluster through a particular node or nodes. For this case, the gateways just need to be scheduled to the desired nodes, and the application pods/namespaces configured to use those gateways. ## Concepts[​](#concepts) ### Egress gateway[​](#egress-gateway) An egress gateway acts as a transit pod for the outbound application traffic that is configured to use it. As traffic leaving the cluster passes through the egress gateway, its source IP is changed to that of the egress gateway pod, and the traffic is then forwarded on. ### Source IP[​](#source-ip) When an outbound application flow leaves the cluster, its IP packets will have a source IP. Normally this is the pod IP of the pod that originated the flow, or the node IP of the node hosting that pod. It will be the **node IP** if the pod IP came from an [IP pool](https://docs.tigera.io/calico-cloud/reference/resources/ippool) with `natOutgoing: true`, and the **pod IP** if not. (Assuming no other CNI plugin has been configured to NAT outgoing traffic.) With an egress gateway involved that is all still true, except that now it's the egress gateway that counts, instead of the original application pod. So the flow will have the egress gateway's **node IP**, if the egress gateway's pod IP came from an [IP pool](https://docs.tigera.io/calico-cloud/reference/resources/ippool) with `natOutgoing: true`, and the egress gateway's **pod IP** otherwise. ### Control the use of egress gateways[​](#control-the-use-of-egress-gateways) If a cluster ascribes special meaning to traffic flowing through egress gateways, it will be important to control when cluster users can configure their pods and namespaces to use them, so that non-special pods cannot impersonate the special meaning. If namespaces in a cluster can only be provisioned by cluster admins, one option is to enable egress gateway function only on a per-namespace basis. Then only cluster admins will be able to configure any egress gateway usage. Otherwise -- if namespace provisioning is open to users in general, or if it's desirable for egress gateway function to be enabled both per-namespace and per-pod -- a [Kubernetes admission controller](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/) will be needed. This is a task for each deployment to implement for itself, but possible approaches include the following. 1. Decide whether a given Namespace or Pod is permitted to use egress annotations at all, based on other details of the Namespace or Pod definition. 2. Evaluate egress annotation selectors to determine the egress gateways that they map to, and decide whether that usage is acceptable. 3. Impose the cluster's own bespoke scheme for a Namespace or Pod to identify the egress gateways that it wants to use, less general than Calico Cloud's egress annotations. Then the admission controller would police those bespoke annotations (that that cluster's users could place on Namespace or Pod resources) and either reject the operation in hand, or allow it through after adding the corresponding Calico Cloud egress annotations. ### Policy enforcement for flows via an egress gateway[​](#policy-enforcement-for-flows-via-an-egress-gateway) For an outbound connection from a client pod, via an egress gateway, to a destination outside the cluster, there is more than one possible enforcement point for policy: The path of the traffic through policy is as follows: 1. Packet leaves the client pod and passes through its egress policy. 2. The packet is encapsulated by the client pod's host and sent to the egress gateway 3. The encapsulated packet is sent from the host to the egress gateway pod. 4. The egress gateway pod de-encapsulates the packet and sends the packet out again with its own address. 5. The packet leaves the egress gateway pod through its egress policy. To ensure correct operation, (as of v3.15) the encapsulated traffic between host and egress gateway is auto-allowed by Calico Cloud and other ingress traffic is blocked. That means that there are effectively two places where policy can be applied: 1. on egress from the client pod 2. on egress from the egress gateway pod (see limitations below). The policy applied at (1) is the most powerful since it implicitly sees the original source of the traffic (by virtue of being attached to that original source). It also sees the external destination of the traffic. Since an egress gateway will never originate its own traffic, one option is to rely on policy applied at (1) and to allow all traffic to at (2) (either by applying no policy or by applying an "allow all"). Alternatively, for maximum "defense in depth" applying policy at both (1) and (2) provides extra protection should the policy at (1) be disabled or bypassed by an attacker. Policy at (2) has the following limitations: - [Domain-based policy](https://docs.tigera.io/calico-cloud/network-policy/domain-based-policy) is not supported at egress from egress gateways. It will either fail to match the expected traffic, or it will work intermittently if the egress gateway happens to be scheduled to the same node as its clients. This is because any DNS lookup happens at the client pod. By the time the policy reaches (2) the DNS information is lost and only the IP addresses of the traffic are available. - The traffic source will appear to be the egress gateway pod, the source information is lost in the address translation that occurs inside the egress gateway pod. That means that policies at (2) will usually take the form of rules that match only on destination port and IP address, either directly in the rule (via a CIDR match) or via a (non-domain based) NetworkSet. Matching on source has little utility since the IP will always be the egress gateway and the port of translated traffic is not always preserved. > **SECONDARY:** Since v3.15.0, Calico Cloud also sends health probes to the egress gateway pods from the nodes where their clients are located. In iptables mode, this traffic is auto-allowed at egress from the host and ingress to the egress gateway. In eBPF mode, the probe traffic can be blocked by policy, so you must ensure that this traffic allowed; this should be fixed in an upcoming patch release. ## Before you begin[​](#before-you-begin) **Unsupported** - GKE **Required** - Calico CNI - Open port UDP 4790 on the host ## How to[​](#how-to) - [Enable egress gateway support](#enable-egress-gateway-support) - [Provision an egress IP pool](#provision-an-egress-ip-pool) - [Deploy a group of egress gateways](#deploy-a-group-of-egress-gateways) - [Configure iptables backend for egress gateways](#configure-iptables-backend-for-egress-gateways) - [Configure namespaces and pods to use egress gateways](#configure-namespaces-and-pods-to-use-egress-gateways) - [Optionally enable ECMP load balancing](#optionally-enable-ecmp-load-balancing) - [Verify the feature operation](#verify-the-feature-operation) - [Control the use of egress gateways](#control-the-use-of-egress-gateways) - [Upgrade egress gateways](#upgrade-egress-gateways) ### Enable egress gateway support[​](#enable-egress-gateway-support) In the default **FelixConfiguration**, set the `egressIPSupport` field to `EnabledPerNamespace` or `EnabledPerNamespaceOrPerPod`, according to the level of support that you need in your cluster. For support on a per-namespace basis only: ```bash kubectl patch felixconfiguration default --type='merge' -p \ '{"spec":{"egressIPSupport":"EnabledPerNamespace"}}' ``` Or for support both per-namespace and per-pod: ```bash kubectl patch felixconfiguration default --type='merge' -p \ '{"spec":{"egressIPSupport":"EnabledPerNamespaceOrPerPod"}}' ``` > **SECONDARY:** > > - `egressIPSupport` must be the same on all cluster nodes, so you should set them only in the `default` FelixConfiguration resource. > - The operator automatically enables the required policy sync API in the FelixConfiguration. ### Provision an egress IP pool[​](#provision-an-egress-ip-pool) Provision a small IP Pool with the range of source IPs that you want to use for a particular application when it connects to an external service. For example: ```bash kubectl apply -f - < **SECONDARY:** This setting is not specific to egress gateway. In some cases where nodes happen to be in the same subnet, setting the value to `Never`will work the same as `Always`. It all depends on the hop from the client node to the egress gateway node. For example, if the client nodes are in the same AWS subnet, and you are using `Always` because some of the nodes are in different subnets, then `Never` will work for the egress IP Pool when the client and gateway nodes are in the same subnet. ### Deploy a group of egress gateways[​](#deploy-a-group-of-egress-gateways) Use an egress gateway custom resource to deploy a group of egress gateways, using the egress IP Pool. ```bash kubectl apply -f - < **SECONDARY:** When deploying egress gateway in a non-default namespace on OpenShift, the namespace needs to be set privileged by adding the following to the namespace: > > ##### Label[​](#label) > > ```text > openshift.io/run-level: "0" > pod-security.kubernetes.io/enforce: privileged > pod-security.kubernetes.io/enforce-version: latest > ``` > > ##### Annotation[​](#annotation) > > ```text > security.openshift.io/scc.podSecurityLabelSync: "false" > ``` Where: - It is advisable to have more than one egress gateway per group, so that the egress IP function continues if one of the gateways crashes or needs to be restarted. When there are multiple gateways in a group, outbound traffic from the applications using that group is load-balanced across the available gateways. The number of `replicas` specified must be less than or equal to the number of free IP addresses in the IP Pool. - IPPool can be specified either by its name (e.g. `-name: egress-ippool-1`) or by its CIDR (e.g. `-cidr: 10.10.10.0/31`). - The labels are arbitrary. You can choose whatever names and values are convenient for your cluster's Namespaces and Pods to refer to in their egress selectors. If labels are not specified, a default label `projectcalico.org/egw`:`name` will be added by the Tigera Operator. - icmpProbe may be used to specify the Probe IPs, ICMP interval and timeout in seconds. `ips` if set, the egress gateway pod will probe each IP periodically using an ICMP ping. If all pings fail then the egress gateway will report non-ready via its health port. `intervalSeconds` controls the interval between probes. `timeoutSeconds` controls the timeout before reporting non-ready if no probes succeed. ```yaml icmpProbe: ips: - probeIP - probeIP timeoutSeconds: 20 intervalSeconds: 10 ``` - httpProbe may be used to specify the Probe URLs, HTTP interval and timeout in seconds. `urls` if set, the egress gateway pod will probe each external service periodically. If all probes fail then the egress gateway will report non-ready via its health port. `intervalSeconds` controls the interval between probes. `timeoutSeconds` controls the timeout before reporting non-ready if all probes are failing. ```yaml httpProbe: urls: - probeURL - probeURL timeoutSeconds: 30 intervalSeconds: 10 ``` - Please refer to the [operator reference docs](https://docs.tigera.io/calico-cloud/reference/installation/api) for details about the egress gateway resource type. The health port `8080` is used by: - The Kubernetes `readinessProbe` to expose the status of the egress gateway pod (and any ICMP/HTTP probes). - Remote pods to check if the egress gateway is "ready". Only "ready" egress gateways will be used for remote client traffic. This traffic is automatically allowed by Calico Cloud and no policy is required to allow it. Calico Cloud only sends probes to egress gateway pods that have a named "health" port. This ensures that during an upgrade, health probes are only sent to upgraded egress gateways. ### Deploy on a RKE2 CIS-hardened cluster[​](#deploy-on-a-rke2-cis-hardened-cluster) If you are deploying `egress-gateway` on a RKE2 CIS-hardened cluster, its `PodSecurityPolicies` restrict the `securityContext` and `volumes` required by egress gateway. When deploying using the egress gateway custom resource, the Tigera Operator sets up `PodSecurityPolicy`, `Role`, `RoleBinding` and associated `ServiceAccount`. ### Configure iptables backend for egress gateways[​](#configure-iptables-backend-for-egress-gateways) The Tigera Operator configures egress gateways to use the same iptables backend as `calico-node`. To modify the iptables backend for egress gateways, you must change the `iptablesBackend` field in the [Felix configuration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig). ### Configure IP autodetection for dual-ToR clusters.[​](#configure-ip-autodetection-for-dual-tor-clusters) If you plan to use Egress Gateways in a [dual-ToR cluster](https://docs.tigera.io/calico-cloud/networking/configuring/dual-tor), you must also adjust the cnx-node IP auto-detection method to pick up the stable IP, for example using the `interface: lo` setting (The default first-found setting skips over the lo interface). This can be configured via the Calico Cloud [Installation resource](https://docs.tigera.io/calico-cloud/reference/installation/api#nodeaddressautodetection). ### Configure namespaces and pods to use egress gateways[​](#configure-namespaces-and-pods-to-use-egress-gateways) You can configure namespaces and pods to use an egress gateway by: - annotating the namespace or pod - applying an egress gateway policy to the namespace or pod. Using an egress gateway policy is more complicated, but it allows advanced use cases. #### Configure a namespace or pod to use an egress gateway (annotation method)[​](#configure-a-namespace-or-pod-to-use-an-egress-gateway-annotation-method) In a Calico Cloud deployment, the Kubernetes namespace and pod resources honor annotations that tell that namespace or pod to use particular egress gateways. These annotations are selectors, and their meaning is "the set of pods, anywhere in the cluster, that match those selectors". So, to configure all the pods in a namespace to use the egress gateways that are labelled with `egress-code: red`, you would annotate that namespace like this: ```bash kubectl annotate ns egress.projectcalico.org/selector="egress-code == 'red'" ``` By default, that selector can only match egress gateways in the same namespace. To select gateways in a different namespace, specify a `namespaceSelector` annotation as well, like this: ```bash kubectl annotate ns egress.projectcalico.org/namespaceSelector="projectcalico.org/name == 'default'" ``` Egress gateway annotations have the same [syntax and range of expressions](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy#selector) as the selector fields in Calico Cloud [network policy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy#entityrule). To configure a specific Kubernetes Pod to use egress gateways, specify the same annotations when creating the pod. For example: ```bash kubectl apply -f - < **SECONDARY:** CIDRs specified in rules in an egress gateway policy are matched in Longest Prefix Match(LPM) fashion. Calico Cloud rejects egress gateway policies that do any of the following: - The policy has no rule that specifies a gateway or a destination - The policy has a rule with empty `selector` or `namespaceSelector` fields. - The policy has two or more rules with the same destination. To configure all the pods in a namespace to use an egress gateway policy named `egw-policy1`, you could annotate the namespace like this: ```bash kubectl annotate ns egress.projectcalico.org/egressGatewayPolicy="egw-policy1" ``` To configure a specific Kubernetes pod to use the same policy, specify the same annotations when creating the pod. For example: ```bash kubectl apply -f - < **WARNING:** You must create the egress gateway policy before you apply it to a namespace or pod. If you attempt to apply an egress gateway policy that has not been created, Calico Cloud will block all traffic from the namespace or pod. ### Optionally enable ECMP load balancing[​](#optionally-enable-ecmp-load-balancing) If you are provisioning multiple egress gateways for a given client pod, and you want traffic from that client to load balance across the available gateways, set the `fib_multipath_hash_policy` [sysctl](https://sysctl-explorer.net/net/ipv4/fib_multipath_hash_policy/) to allow that: ```bash sudo sysctl -w net.ipv4.fib_multipath_hash_policy=1 ``` You will need this on each node with clients that you want to load balance across multiple egress gateways. ### Verify the feature operation[​](#verify-the-feature-operation) To verify the feature operation, cause the application pod to initiate a connection to a server outside the cluster, and observe -- for example using tcpdump -- the source IP of the connection packet as it reaches the server. > **SECONDARY:** In order for such a connection to complete, the server must know how to route back to the egress gateway's IP. By way of a concrete example, you could use netcat to run a test server outside the cluster; for example: ```bash docker run --net=host --privileged subfuzion/netcat -v -l -k -p 8089 ``` Then provision an egress IP Pool, and egress gateways, as above. Then deploy a pod, with egress annotations as above, and with any image that includes netcat, for example: ```bash kubectl apply -f - < -n -- nc 8089 ` should be the IP address of the netcat server. Then, if you check the logs or output of the netcat server, you should see: ```text Connection from received ``` with `` being one of the IPs of the egress IP pool that you provisioned. ## Upgrade egress gateways[​](#upgrade-egress-gateways) From v3.16, egress gateway deployments are managed by the Tigera Operator. - When upgrading from a pre-v3.16 release, no automatic upgrade will occur. To upgrade a pre-v3.16 egress gateway deployment, create an equivalent EgressGateway resource with the same namespace and the same name as mentioned [above](#deploy-a-group-of-egress-gateways); the operator will then take over management of the old Deployment resource, replacing it with the upgraded version. - Use `kubectl apply` to create the egress gateway resource. Tigera Operator will read the newly created resource and wait for the other Calico Cloud components to be upgraded. Once the other Calico Cloud components are upgraded, Tigera Operator will upgrade the existing egress gateway deployment with the new image. By default, upgrading egress gateways will sever any connections that are flowing through them. To minimise impact, the egress gateway feature supports some advanced options that give feedback to affected pods. For more details see the [egress gateway maintenance guide](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-maintenance). ## Additional resources[​](#additional-resources) Please see also: - The `egressIP...` fields of the [FelixConfiguration resource](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig#spec). - [Additional configuration for egress gateway maintenance](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-maintenance) ### Configure egress gateways, AWS ## Big picture[​](#big-picture) Control the source IP address seen by external services/appliances by routing the traffic from certain pods through egress gateways. Use native VPC subnet IP addresses for the egress gateways so that the IPs are valid in the AWS fabric. ## Value[​](#value) Controlling the source IP seen when traffic leaves the cluster allows groups of pods to be identified by external firewalls, appliances and services (even as the groups are scaled up/down or pods restarted). Calico Cloud controls the source IP by directing traffic through one or more "egress gateway" pods, which change the source IP of the traffic to their own IP. The egress gateways used can be chosen at the pod or namespace scope allowing for flexibility in how the cluster is seen from outside. In AWS, egress gateway source IP addresses are chosen from an IP pool backed by a [VPC subnet](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Subnets.html) using Calico Cloud IPAM. Calico Cloud IPAM allows the IP addresses to be precisely controlled, this allows for static configuration of external appliances. Using an IP pool backed by a VPC subnet allows Calico Cloud to configure the AWS fabric to route traffic to and from the egress gateway using its own IP address. ## Concepts[​](#concepts) ### CIDR notation[​](#cidr-notation) This article assumes that you are familiar with network masks and CIDR notation. - CIDR notation is defined in [RFC4632](https://datatracker.ietf.org/doc/html/rfc4632). - The [Wikipedia article on CIDR notation](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing#CIDR_notation) provides a good reference. ### AWS-backed IP pools[​](#aws-backed-ip-pools) Calico Cloud supports IP pools that are backed by the AWS fabric. Workloads that use an IP address from an AWS-backed pool can communicate on the AWS network using their own IP address and AWS will route their traffic to/from their host without changing the IP address. Pods that use an IP address from an AWS-backed pool may also be [assigned an AWS Elastic IP via a pod annotation](#add-aws-elastic-ips-to-the-egress-gateway-deployment) . Elastic IPs used in this way have the normal AWS semantics: when accessing resources inside the AWS network, the workload's private IP (from the IP pool) is used. When accessing resources outside the AWS network, AWS translates the workload's IP to the Elastic IP. Elastic IPs also allow for incoming requests from outside the AWS fabric, direct to the workload. In overview, the AWS-backed IP Pools feature works as follows: - An IP pool is created with its `awsSubnetID` field set to the ID of a VPC subnet. This "AWS-backed" IP pool's CIDR must be contained within the VPC subnet's CIDR. > **WARNING:** You must ensure that the CIDR(s) used for AWS-backed IP pool(s) are reserved in the AWS fabric. For example, by creating a dedicated VPC subnet for Calico Cloud. If the CIDR is not reserved; both Calico Cloud and AWS may try to assign the same IP address, resulting in a conflict. - Since they are a limited resource, Calico Cloud IPAM does not use AWS-backed pools by default. To request an AWS-backed IP address, a pod must have a resource request: ```yaml spec: containers: - ... resources: requests: projectcalico.org/aws-secondary-ipv4: 1 limits: projectcalico.org/aws-secondary-ipv4: 1 ``` Calico Cloud manages the `projectcalico.org/aws-secondary-ipv4` capacity on the Kubernetes Node resource, ensuring that Kubernetes will not try to schedule too many AWS-backed workloads to the same node. Only AWS-backed pods are limited in this way; there is no limit on the number of non-AWS-backed pods. - When the CNI plugin spots such a resource request, it will choose an IP address from an AWS-backed pool. Only pools with VPC subnets in the availability zone of the host are considered. - When Felix, Calico Cloud's per-host agent spots a local workload with an AWS-backed address it tries to ensure that the IP address of the workload is assigned to the host in the AWS fabric. If need be, it will create a new [secondary ENI](#secondary-elastic-network-interfaces-enis) device and attach it to the host to house the IP address. Felix supports two modes for assigning secondary ENIs: **ENI-per-workload** mode (added in v3.13) and **Secondary-IP-per-workload** mode. These modes are described [below](#secondary-elastic-network-interfaces-enis). - If the pod has one or more AWS Elastic IPs listed in the `cni.projectcalico.org/awsElasticIPs` pod annotation, Felix will try to ensure that *one* of the Elastic IPs is assigned to the pod's private IP address in the AWS fabric. (Specifying multiple Elastic IPs is useful for multi-pod deployments; ensuring that each pod in the deployment gets one of the IPs.) ### Egress gateway[​](#egress-gateway) An egress gateway acts as a transit pod for the outbound application traffic that is configured to use it. As traffic leaving the cluster passes through the egress gateway, its source IP is changed to that of the egress gateway pod, and the traffic is then forwarded on. ### Source IP[​](#source-ip) When an outbound application flow leaves the cluster, its IP packets will have a source IP. This begins as the pod IP of the pod that originated the flow, then: - *If no egress gateway is configured* and the pod IP came from an [IP pool](https://docs.tigera.io/calico-cloud/reference/resources/ippool) with `natOutgoing: true`, the node hosting the pod will change the source IP to its own as the traffic leaves the host. This allows the pod to communicate with external service even though the external network is unaware of the pod's IP. - *If the pod is configured with an egress gateway*, the traffic is first forwarded to the egress gateway, which changes the source IP to its own and then sends the traffic on. To function correctly, egress gateways should have IPs from an IP pool with `natOutgoing: false`, meaning their host forwards the packet onto the network without changing the source IP again. Since the egress gateway's IP is visible to the underlying network fabric, the fabric must be configured to know about the egress gateway's IP and to send response traffic back to the same host. ### AWS VPCs and subnets[​](#aws-vpcs-and-subnets) An AWS VPC is a virtual network that is, by default, logically isolated from other VPCs. Each VPC has one or more (often large) CIDR blocks associated with it (for example `10.0.0.0/16`). In general, VPC CIDRs may overlap, but only if the VPCs remain isolated. AWS allows VPCs to be peered with each other through VPC Peerings. VPCs can only be peered if *none* of their associated CIDRs overlap. Each VPC has one or more VPC subnets associated with it, each subnet owns a non-overlapping part of one of the VPC's CIDR blocks. Each subnet is associated with a particular availability zone. Instances in one availability zone can only use IP addresses from subnets in that zone. Unfortunately, this adds some complexity to managing egress gateways IP addresses: much of the configuration must be repeated per-AZ. ### AWS VPC and DirectConnect peerings[​](#aws-vpc-and-directconnect-peerings) AWS [VPC Peerings](https://docs.aws.amazon.com/vpc/latest/peering/vpc-peering-basics.html) allow multiple VPCs to be connected together. Similarly, [DirectConnect](https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html) allows external datacenters to be connected to an AWS VPC. Peered VPCs and datacenters communicate using private IPs as if they were all on one large private network. By using AWS-backed IP pools, egress gateways can be assigned private IPs allowing them to communicate without NAT within the same VPC, with peered VPCs, and, with peered datacenters. ### Secondary Elastic Network Interfaces (ENIs)[​](#secondary-elastic-network-interfaces-enis) Elastic network interfaces are network interfaces that can be added and removed from an instance dynamically. Each ENI has a primary IP address from the VPC subnet that it belongs to, and it may also have one or more secondary IP addresses, chosen for the same subnet. While the primary IP address is fixed and cannot be changed, the secondary IP addresses can be added and removed at runtime. To arrange for AWS to route traffic to and from egress gateways, Calico Cloud adds *secondary* Elastic Network Interfaces (ENIs) to the host. Calico Cloud supports two modes for provisioning the secondary ENIs. The table below describes the trade-offs between **ENI-per-workload** and **Secondary-IP-per-workload** modes: | **ENI-per-workload** (since v3.13) | **Secondary-IP-per-workload** | | ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | | One secondary ENI is attached for each AWS-backed workload. | Secondary ENIs are shared, multiple workloads per ENI. | | Supports one AWS-backed workload per secondary ENI. | Supports 2-49 AWS-backed workloads per secondary ENI (depending on instance type). | | ENI Primary IP is set to Workload's IP. | ENI Primary IP chosen from dedicated "host secondary" IP pools. | | Makes best use of AWS IP space, no need to reserve IPs for hosts. | Requires "host secondary" IPs to be reserved. These cannot be used for workloads. | | ENI deleted when workload deleted. | ENI retained (ready for next workload to be scheduled). | | Slower to handle churn/workload mobility. (Creating ENI is slower than assigning IP.) | Faster at handling churn/workload mobility. | The number of ENIs that an instance can support and the number of secondary IPs that each ENI can support depends on the instance type according to [this table](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI). Note: the table lists the total number of network interfaces and IP addresses but the first interface on the host (the primary interface) and, in Secondary-IP-per-workload mode, the first IP of each interface (its primary IP) cannot be used for egress gateways. The primary interface cannot be used for egress gateways because it belongs to the VPC subnet that is in use for Kubernetes hosts; this means that a planned egress gateway IP could get used by AWS as the primary IP of an instance (for example when scaling up the cluster). ## Before you begin[​](#before-you-begin) **Required** - Calico CNI - Open port UDP 4790 on the host **Not Supported** - Amazon VPC CNI Calico Cloud CNI and IPAM is required. The ability to control the egress gateway’s IP is a feature of Calico Cloud CNI and IPAM. AWS VPC CNI does not support that feature, so it is incompatible with egress gateways. ## How to[​](#how-to) - [Configure IP autodetection](#configure-ip-autodetection) - [Ensure Kubernetes VPC has free CIDR range](#ensure-kubernetes-vpc-has-free-cidr-range) - [Create dedicated VPC subnets](#create-dedicated-vpc-subnets) - [Configure AWS IAM roles](#configure-aws-iam-roles) - [Configure IP reservations for each VPC subnet](#configure-ip-reservations-for-each-vpc-subnet) - [Enable egress gateway support](#enable-egress-gateway-support) - [Enable AWS-backed IP pools](#enable-aws-backed-ip-pools) - [Configure IP pools backed by VPC subnets](#configure-ip-pools-backed-by-vpc-subnets) - [Deploy a group of egress gateways](#deploy-a-group-of-egress-gateways) - [Configure iptables backend for egress gateways](#configure-iptables-backend-for-egress-gateways) - [Configure namespaces and pods to use egress gateways](#configure-namespaces-and-pods-to-use-egress-gateways) - [Optionally enable ECMP load balancing](#optionally-enable-ecmp-load-balancing) - [Verify the feature operation](#verify-the-feature-operation) - [Control the use of egress gateways](#control-the-use-of-egress-gateways) - [Policy enforcement for flows via an egress gateway](#policy-enforcement-for-flows-via-an-egress-gateway) - [Upgrade egress gateways](#upgrade-egress-gateways) ### Configure IP autodetection[​](#configure-ip-autodetection) Since this feature adds additional network interfaces to nodes, it is important to configure Calico Cloud to autodetect the correct primary interface to use for normal pod-to-pod traffic. Otherwise, Calico Cloud may autodetect a newly-added secondary ENI as the main interface, causing an outage. For EKS clusters, the default IP autodetection method is `can-reach=8.8.8.8`, which will choose the interface with a route to `8.8.8.8`; this is typically the interface with a default route, which will be the correct (primary) ENI. (Calico Cloud ensures that the secondary ENIs do not have default routes in the main routing table.) For other AWS clusters, Calico Cloud may default to `firstFound`, which is **not** suitable. To examine the autodetection method, check the operator's installation resource: ```bash $ kubectl get installations.operator.tigera.io -o yaml default ``` ```yaml apiVersion: operator.tigera.io/v1 kind: Installation metadata: ... name: default ... spec: calicoNetwork: ... nodeAddressAutodetectionV4: firstFound: true ... ``` If `nodeAddressAutodetectionV4` is set to `firstFound: true` or is not specified, then you must change it to another method by editing the resource. The NodeAddressAutodetection options, `canReach` and `cidrs` are suitable. See [Installation reference](https://docs.tigera.io/calico-cloud/reference/installation/api). If using the `cidrs` option, set the CIDRs list to include only the CIDRs from which your primary ENI IPs are chosen (do not include the dedicated VPC subnets chosen below). ### Ensure Kubernetes VPC has free CIDR range[​](#ensure-kubernetes-vpc-has-free-cidr-range) For egress gateways to be useful in AWS, we want to assign them IP addresses from a VPC subnet that is in the same AZ as their host. To avoid clashes between AWS IP allocations and Calico Cloud IP allocations, it is important that the range of IP addresses assigned to Calico Cloud IP pools is not used by AWS for automatic allocations. In this guide we assume that you have created a dedicated VPC subnet per Availability Zone (AZ) that is reserved for Calico Cloud and configured not to be used as the default subnet for the AZ. If you are creating your cluster and VPC from scratch, plan to subdivide the VPC CIDR into (at least) two VPC subnets per AZ. One VPC subnet for the Kubernetes (and any other) hosts and one VPC subnet for egress gateways. (The next section explains the sizing requirements for the egress gateway subnets.) If you are adding this feature to an existing cluster, you may find that the existing VPC subnets already cover the entire VPC CIDR, making it impossible to create a new subnet. If that is the case, you can make more room by adding a second CIDR to the VPC that is large enough for the new subnets. For information on adding a secondary CIDR range to a VPC, see [this guide](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Subnets.html#vpc-resize). ### Create dedicated VPC subnets[​](#create-dedicated-vpc-subnets) Calico Cloud requires a dedicated VPC subnet in each AWS availability zone that you wish to deploy egress gateways. The subnet must be dedicated to Calico Cloud so that AWS will not use IP addresses from the subnet for other purposes (as this could clash with an egress gateway's IP). When creating the subnet you should configure it not to be used for instances. Some IP addresses from the dedicated subnet are reserved for AWS and Calico Cloud internal use: - The first four IP addresses in the subnet cannot be used. These are [reserved by AWS for internal use](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Subnets.html#vpc-sizing-ipv4). - Similarly, the last IP in the subnet (the broadcast address) cannot be used. - *In **Secondary-IP-per-workload** mode*, Calico Cloud requires one IP address from the subnet per secondary ENI that it provisions (for use as the primary IP address of the ENI). In **ENI-per-workload** mode, this is not required. **Tab: ENI-per-workload** Example for **ENI-per-workload** mode: - You anticipate having up to 30 instances running in each availability zone (AZ). - You intend to use `t3.large` instances, [these are limited to](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI) 3 ENIs per host. - So, each host can accept 2 secondary ENIs, each of which can handle one egress gateway. - With 2 ENIs per node and 30 nodes, the part of the cluster in this AZ could handle up to `30 * 2 = 60` egress gateways. - AWS reserves 5 IPs from the AWS subnet for internal use, no "host secondary IPs" need to be reserved in this mode. - Since VPC subnets are allocated by CIDR, a `/25` subnet containing 128 IP addresses would comfortably fit the 5 reserved IPs as well as the 60 possible gateways (with headroom for more nodes to be added later). **Tab: Secondary-IP-per-workload** Example for **Secondary-IP-per-workload** mode: - You anticipate having up to 30 instances running in each availability zone (AZ). - You intend to use `t3.large` instances, [these are limited to](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI) 3 ENIs per host (one of which is the primary) and each ENI can handle 12 IP addresses, (one of which is the primary). - So, each host can accept 2 secondary ENIs and each secondary ENI could handle 11 egress gateway pods. - Each in-use secondary ENI requires one IP from the VPC subnet (up to 60 in this case) and AWS requires 5 IPs to be reserved so that's up to 65 IPs reserved in total. - With 2 ENIs and 11 IPs per ENI, the part of the cluster in this AZ could handle up to `30 * 2 * 11 = 660` egress gateways. - Since VPC subnets are allocated by CIDR, a `/22` subnet containing 1024 IP addresses would comfortably fit the 65 reserved IPs as well as the 660 possible gateways. Calico Cloud allocates ENIs on-demand so each instance will only claim one of those reserved IP addresses when the first egress gateway is assigned to that node. It will only claim its second IP when that ENI becomes full and then an extra egress gateway is provisioned. ### Configure AWS IAM roles[​](#configure-aws-iam-roles) To provision the required AWS resources, each calico-node pod in your cluster requires the following IAM permissions to be granted. The permissions can be granted to the node IAM Role itself, or by using the AWS [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) feature to grant the permissions to the `calico-node` service account. - DescribeInstances - DescribeInstanceTypes - DescribeNetworkInterfaces - DescribeSubnets - DescribeTags - CreateTags - AssignPrivateIpAddresses - UnassignPrivateIpAddresses - AttachNetworkInterface - CreateNetworkInterface - DeleteNetworkInterface - DetachNetworkInterface - ModifyNetworkInterfaceAttribute The above permissions are similar to those used by the AWS VPC CNI (since both CNIs need to provision the same kinds of resources). In addition, to support elastic IPs, each calico-node also requires the following permissions: - DescribeAddresses - AssociateAddress - DisassociateAddress ### Configure AWS Security Group rules[​](#configure-aws-security-group-rules) To allow egress gateway traffic into the egress gateway pod's host from the client, the ingress rules of the security group need to be updated. A rule to allow all packets from within the security group must be added to the inbound rules. ### Configure IP reservations for each VPC subnet[​](#configure-ip-reservations-for-each-vpc-subnet) Since the first four IP addresses and the last IP address in a VPC subnet cannot be used, it is important to prevent Calico Cloud from *trying* to use them. For each VPC subnet that you plan to use, ensure that you have an entry in an [IP reservation](https://docs.tigera.io/calico-cloud/reference/resources/ipreservation) for its first four IP addresses and its final IP address. For example, if your chosen VPC subnets are `100.64.0.0/22` and `100.64.4.0/22`, you could create the following `IPReservation` resource, which covers both VPC subnets (if you're not familiar with CIDR notation, replacing the `/22` of the original subnet with `/30` is a shorthand for "the first four IP addresses"): ```yaml apiVersion: projectcalico.org/v3 kind: IPReservation metadata: name: aws-ip-reservations spec: reservedCIDRs: - 100.64.0.0/30 - 100.64.3.255 - 100.64.4.0/30 - 100.64.7.255 ``` ### Enable egress gateway support[​](#enable-egress-gateway-support) In the default **FelixConfiguration**, set the `egressIPSupport` field to `EnabledPerNamespace` or `EnabledPerNamespaceOrPerPod`, according to the level of support that you need in your cluster. For support on a per-namespace basis only: ```bash kubectl patch felixconfiguration default --type='merge' -p \ '{"spec":{"egressIPSupport":"EnabledPerNamespace"}}' ``` Or for support both per-namespace and per-pod: ```bash kubectl patch felixconfiguration default --type='merge' -p \ '{"spec":{"egressIPSupport":"EnabledPerNamespaceOrPerPod"}}' ``` > **SECONDARY:** > > - `egressIPSupport` must be the same on all cluster nodes, so you should set them only in the `default` FelixConfiguration resource. > - The operator automatically enables the required policy sync API in the FelixConfiguration. ### Enable AWS-backed IP pools[​](#enable-aws-backed-ip-pools) **Tab: ENI-per-workload** To enable **ENI-per-workload** mode, in the default **FelixConfiguration**, set the `awsSecondaryIPSupport` field to `EnabledENIPerWorkload`: ```bash kubectl patch felixconfiguration default --type='merge' -p \ '{"spec":{"awsSecondaryIPSupport":"EnabledENIPerWorkload"}}' ``` **Tab: Secondary-IP-per-workload** To enable **Secondary-IP-per-workload** mode, set the field to `Enabled` (the name `Enabled` predates the addition of the **ENI-per-workload** mode): ```bash kubectl patch felixconfiguration default --type='merge' -p \ '{"spec":{"awsSecondaryIPSupport":"Enabled"}}' ``` You can verify that the setting took effect by examining the Kubernetes Node resources: ```bash kubectl describe node ``` Should show the new `projectcalico.org/aws-secondary-ipv4` capacity (in the Allocated Resources section). #### Changing modes[​](#changing-modes) You can change between the two modes by: - Ensuring that the number of egress gateways on every node is within the limits of the particular mode. i.e. when switching to **ENI-per-workload** mode, the number of egress gateways must be less than or equal to the number of secondary ENIs that your instances can handle. - Editing the setting (using the patch commands above, for example). Changing the mode will cause disruption as ENIs must be removed and re-added. ### Configure IP pools backed by VPC subnets[​](#configure-ip-pools-backed-by-vpc-subnets) **Tab: ENI-per-workload** In **ENI-per-workload** mode, IP pools are (only) used to subdivide the VPC subnets into small pools used for particular groups of egress gateways. These IP Pools must have: - `awsSubnetID` set to the ID of the relevant VPC subnet. This activates the AWS-backed IP feature for these pools. - `allowedUse` set to `["Workload"]` to tell Calico Cloud IPAM to use those pools for the egress gateway workloads. - `vxlanMode` and `ipipMode` set to `Never` to disable encapsulation for the egress gateway pods. (`Never` is the default if these fields are not specified.) - `blockSize` set to 32. This aligns Calico Cloud IPAM with the behaviour of the AWS fabric. - `disableBGPExport` set to `true`. This prevents routing conflicts if your cluster is using IPIP or BGP networking. It's also recommended to: - Set `nodeSelector` to `"!all()"`. This prevents Calico Cloud IPAM from using the pool automatically. It will only be used for workloads that explicitly name it in the `cni.projectcalico.org/ipv4pools` annotation. Continuing the example above, with VPC subnets - `100.64.0.0/22` in, say, availability zone west-1 and id `subnet-000000000000000001` - `100.64.4.0/22` in, say, availability zone west-2 and id `subnet-000000000000000002` And, assuming that there are two clusters of egress gateways "red" and "blue" (which in turn serve namespaces "red" and "blue"), one way to structure the IP pools is to have one IP pool for each group of egress gateways in each subnet. Then, if a particular egress gateway from the egress gateway cluster is scheduled to one AZ or the other, it will take an IP from the appropriate pool. For the "west-1" availability zone: - IP pool "egress-red-west-1", CIDR `100.64.0.4/30` (the first non-reserved /30 CIDR in the VPC subnet). These addresses will be used for "red" egress gateways in the "west-1" AZ. - IP pool "egress-blue-west-1", CIDR `100.64.0.8/30` (the next 4 IPs from the "west-1" subnet). These addresses will be used for "blue" egress gateways in the "west-1" AZ. For the "west-2" availability zone: - IP pool "egress-red-west-2", CIDR `100.64.4.4/30` (the first non-reserved /30 CIDR in the VPC subnet). These addresses will be used for "red" egress gateways in the "west-2" AZ. - IP pool "egress-blue-west-2", CIDR `100.64.4.8/30` (the next 4 IPs from the "west-2" subnet). These addresses will be used for "blue" egress gateways in the "west-2" AZ. Converting this to `IPPool` resources: ```yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: egress-red-west-1 spec: cidr: 100.64.0.4/30 allowedUses: ['Workload'] awsSubnetID: subnet-000000000000000001 blockSize: 32 nodeSelector: '!all()' disableBGPExport: true --- apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: egress-blue-west-1 spec: cidr: 100.64.0.8/30 allowedUses: ['Workload'] awsSubnetID: subnet-000000000000000001 blockSize: 32 nodeSelector: '!all()' disableBGPExport: true --- apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: egress-red-west-2 spec: cidr: 100.64.4.4/30 allowedUses: ['Workload'] awsSubnetID: subnet-000000000000000002 blockSize: 32 nodeSelector: '!all()' disableBGPExport: true --- apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: egress-blue-west-2 spec: cidr: 100.64.4.8/30 allowedUses: ['Workload'] awsSubnetID: subnet-000000000000000002 blockSize: 32 nodeSelector: '!all()' disableBGPExport: true ``` **Tab: Secondary-IP-per-workload** In **Secondary-IP-per-workload** mode, IP pools are used to subdivide the VPC subnets as follows: - One medium-sized IP pool per-Subnet reserved for Calico Cloud to use for the *primary* IP addresses of its *secondary* ENIs. These pools must have: - `awsSubnetID` set to the ID of the relevant VPC subnet. This activates the AWS-backed IP feature for these pools. - `allowedUse` set to `["HostSecondaryInterface"]` to reserve them for this purpose. - `blockSize` set to 32. This aligns Calico Cloud IPAM with the behaviour of the AWS fabric. - `vxlanMode` and `ipipMode` set to `Never`. (`Never` is the default if these fields are not specified.) - `disableBGPExport` set to `true`. This prevents routing conflicts if your cluster is using IPIP or BGP networking. - Small pools used for particular groups of egress gateways. These must have: - `awsSubnetID` set to the ID of the relevant VPC subnet. This activates the AWS-backed IP feature for these pools. - `allowedUse` set to `["Workload"]` to tell Calico Cloud IPAM to use those pools for the egress gateway workloads. - `vxlanMode` and `ipipMode` set to `Never` to disable encapsulation for the egress gateway pods. (`Never` is the default if these fields are not specified.) - `blockSize` set to 32. This aligns Calico Cloud IPAM with the behaviour of the AWS fabric. - `disableBGPExport` set to `true`. This prevents routing conflicts if your cluster is using IPIP or BGP networking. It's also recommended to: - Set `nodeSelector` to `"!all()"`. This prevents Calico Cloud IPAM from using the pool automatically. It will only be used for workloads that explicitly name it in the `cni.projectcalico.org/ipv4pools` annotation. Continuing the example above, with VPC subnets - `100.64.0.0/22` in, say, availability zone west-1 and id `subnet-000000000000000001` - `100.64.4.0/22` in, say, availability zone west-2 and id `subnet-000000000000000002` And, assuming that there are two clusters of egress gateways "red" and "blue" (which in turn serve namespaces "red" and "blue"), one way to structure the IP pools is to have a "hosts" IP pool in each VPC subnet and one IP pool for each group of egress gateways in each subnet. Then, if a particular egress gateway from the egress gateway cluster is scheduled to one AZ or the other, it will take an IP from the appropriate pool. For the "west-1" availability zone: - IP pool "hosts-west-1", CIDR `100.64.0.0/25` (the first 128 addresses in the "west-1" VPC subnet). - We'll reserve these addresses for hosts to use. - `100.64.0.0/25` covers the addresses from `100.64.0.0` to `100.64.0.127` (but addresses `100.64.0.0` to `100.64.0.3` were reserved above). - IP pool "egress-red-west-1", CIDR `100.64.0.128/30` (the next 4 IPs from the "west-1" subnet). - These addresses will be used for "red" egress gateways in the "west-1" AZ. - IP pool "egress-blue-west-1", CIDR `100.64.0.132/30` (the next 4 IPs from the "west-1" subnet). - These addresses will be used for "blue" egress gateways in the "west-1" AZ. For the "west-2" availability zone: - IP pool "hosts-west-2", CIDR `100.64.4.0/25` (the first 128 addresses in the "west-2" VPC subnet). - `100.64.4.0/25` covers the addresses from `100.64.4.0` to `100.64.4.127` (but addresses `100.64.4.0` to `100.64.4.3` were reserved above). - IP pool "egress-red-west-2", CIDR `100.64.4.128/30` (the next 4 IPs from the "west-2" subnet). - These addresses will be used for "red" egress gateways in the "west-2" AZ. - IP pool "egress-blue-west-2", CIDR `100.64.4.132/30` (the next 4 IPs from the "west-2" subnet). - These addresses will be used for "blue" egress gateways in the "west-2" AZ. Converting this to `IPPool` resources: ```yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: hosts-west-1 spec: cidr: 100.64.0.0/25 allowedUses: ['HostSecondaryInterface'] awsSubnetID: subnet-000000000000000001 blockSize: 32 disableBGPExport: true --- apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: egress-red-west-1 spec: cidr: 100.64.0.128/30 allowedUses: ['Workload'] awsSubnetID: subnet-000000000000000001 blockSize: 32 nodeSelector: '!all()' disableBGPExport: true --- apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: egress-blue-west-1 spec: cidr: 100.64.0.132/30 allowedUses: ['Workload'] awsSubnetID: subnet-000000000000000001 blockSize: 32 nodeSelector: '!all()' disableBGPExport: true --- apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: hosts-west-2 spec: cidr: 100.64.4.0/25 allowedUses: ['HostSecondaryInterface'] awsSubnetID: subnet-000000000000000002 blockSize: 32 disableBGPExport: true --- apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: egress-red-west-2 spec: cidr: 100.64.4.128/30 allowedUses: ['Workload'] awsSubnetID: subnet-000000000000000002 blockSize: 32 nodeSelector: '!all()' disableBGPExport: true --- apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: egress-blue-west-2 spec: cidr: 100.64.4.132/30 allowedUses: ['Workload'] awsSubnetID: subnet-000000000000000002 blockSize: 32 nodeSelector: '!all()' disableBGPExport: true ``` ### Deploy a group of egress gateways[​](#deploy-a-group-of-egress-gateways) Use an egress gateway custom resource to deploy a group of egress gateways. Using the example of the "red" egress gateway cluster, we use several features of Kubernetes and Calico Cloud in tandem to get a cluster of egress gateways that spans both availability zones and uses AWS-backed IP addresses: ```bash kubectl apply -f - < # - # timeoutSeconds: 15 # intervalSeconds: 5 # httpProbe: # urls: # - # - # timeoutSeconds: 30 # intervalSeconds: 10 aws: nativeIP: Enabled template: metadata: labels: egress-code: red spec: nodeSelector: kubernetes.io/os: linux terminationGracePeriodSeconds: 0 topologySpreadConstraints: - maxSkew: 1 topologyKey: "topology.kubernetes.io/zone" whenUnsatisfiable: "DoNotSchedule" labelSelector: matchLabels: egress-code: red EOF ``` - `replicas: 2` tells Kubernetes to schedule two egress gateways in the "red" cluster. - ipPools tells Calico Cloud IPAM to use one of the "red" IP pools: ```yaml ipPools: - name: "egress-red-west-1" - name: "egress-red-west-2" ``` Depending on which AZ the pod is scheduled in, Calico Cloud IPAM will automatically ignore IP pools that are backed by AWS subnets that are not in the local AZ. External services and appliances can recognise "red" traffic because it will all come from the CIDRs of the "red" IP pools. - When nativeIP is enabled, IPPools must be AWS-backed. It also tells Kubernetes to only schedule the gateway to a node with available AWS IP capacity: ```yaml aws: nativeIP: Enabled ``` - The following [topology spread constraint](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/) ensures that Kubernetes spreads the Egress gateways evenly between AZs (assuming that your nodes are labeled with the expected [well-known label](https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone) `topology.kubernetes.io/zone`): ```yaml topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: egress-code: red ``` - The labels are arbitrary. You can choose whatever names and values are convenient for your cluster's Namespaces and Pods to refer to in their egress selectors. If labels are not specified, a default label `projectcalico.org/egw`:`name` will be added by the Tigera Operator. - icmpProbe may be used to specify the Probe IPs, ICMP interval and timeout in seconds. `ips` if set, the egress gateway pod will probe each IP periodically using an ICMP ping. If all pings fail then the egress gateway will report non-ready via its health port. `intervalSeconds` controls the interval between probes. `timeoutSeconds` controls the timeout before reporting non-ready if no probes succeed. ```yaml icmpProbe: ips: - - timeoutSeconds: 20 intervalSeconds: 10 ``` - httpProbe may be used to specify the Probe URLs, HTTP interval and timeout in seconds. `urls` if set, the egress gateway pod will probe each external service periodically. If all probes fail then the egress gateway will report non-ready via its health port. `intervalSeconds` controls the interval between probes. `timeoutSeconds` controls the timeout before reporting non-ready if all probes are failing. ```yaml httpProbe: urls: - - timeoutSeconds: 30 intervalSeconds: 10 ``` - Please refer to the [operator reference docs](https://docs.tigera.io/calico-cloud/reference/installation/api) for details about the egress gateway resource type. > **SECONDARY:** > > - It is advisable to have more than one egress gateway per group, so that the egress IP function continues if one of the gateways crashes or needs to be restarted. When there are multiple gateways in a group, outbound traffic from the applications using that group is load-balanced across the available gateways. The number of `replicas` specified must be less than or equal to the number of free IP addresses in the IP Pool. > - IPPool can be specified either by its name (e.g. `-name: egress-ippool-1`) or by its CIDR (e.g. `-cidr: 10.10.10.0/31`). > - The labels are arbitrary. You can choose whatever names and values are convenient for your cluster's Namespaces and Pods to refer to in their egress selectors. The health port `8080` is used by: > - The Kubernetes `readinessProbe` to expose the status of the egress gateway pod (and any ICMP/HTTP probes). > - Remote pods to check if the egress gateway is "ready". Only "ready" egress gateways will be used for remote client traffic. This traffic is automatically allowed by Calico Cloud and no policy is required to allow it. Calico Cloud only sends probes to egress gateway pods that have a named "health" port. This ensures that during an upgrade, health probes are only sent to upgraded egress gateways. ### Configure iptables backend for egress gateways[​](#configure-iptables-backend-for-egress-gateways) The Tigera Operator configures egress gateways to use the same iptables backend as `calico-node`. To modify the iptables backend for egress gateways, you must change the `iptablesBackend` field in the [Felix configuration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig). ### Configure namespaces and pods to use egress gateways[​](#configure-namespaces-and-pods-to-use-egress-gateways) You can configure namespaces and pods to use an egress gateway by: - annotating the namespace or pod - applying an egress gateway policy to the namespace or pod. Using an egress gateway policy is more complicated, but it allows advanced use cases. #### Configure a namespace or pod to use an egress gateway (annotation method)[​](#configure-a-namespace-or-pod-to-use-an-egress-gateway-annotation-method) In a Calico Cloud deployment, the Kubernetes namespace and pod resources honor annotations that tell that namespace or pod to use particular egress gateways. These annotations are selectors, and their meaning is "the set of pods, anywhere in the cluster, that match those selectors". So, to configure all the pods in a namespace to use the egress gateways that are labelled with `egress-code: red`, you would annotate that namespace like this: ```bash kubectl annotate ns egress.projectcalico.org/selector="egress-code == 'red'" ``` By default, that selector can only match egress gateways in the same namespace. To select gateways in a different namespace, specify a `namespaceSelector` annotation as well, like this: ```bash kubectl annotate ns egress.projectcalico.org/namespaceSelector="projectcalico.org/name == 'default'" ``` Egress gateway annotations have the same [syntax and range of expressions](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy#selector) as the selector fields in Calico Cloud [network policy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy#entityrule). To configure a specific Kubernetes Pod to use egress gateways, specify the same annotations when creating the pod. For example: ```bash kubectl apply -f - < **SECONDARY:** CIDRs specified in rules in an egress gateway policy are matched in Longest Prefix Match(LPM) fashion. Calico Cloud rejects egress gateway policies that do any of the following: - The policy has no rule that specifies a gateway or a destination - The policy has a rule with empty `selector` or `namespaceSelector` fields. - The policy has two or more rules with the same destination. To configure all the pods in a namespace to use an egress gateway policy named `egw-policy1`, you could annotate the namespace like this: ```bash kubectl annotate ns egress.projectcalico.org/egressGatewayPolicy="egw-policy1" ``` To configure a specific Kubernetes pod to use the same policy, specify the same annotations when creating the pod. For example: ```bash kubectl apply -f - < **WARNING:** You must create the egress gateway policy before you apply it to a namespace or pod. If you attempt to apply an egress gateway policy that has not been created, Calico Cloud will block all traffic from the namespace or pod. ### Add AWS Elastic IPs to the egress gateway deployment[​](#add-aws-elastic-ips-to-the-egress-gateway-deployment) To add AWS Elastic IPs to the egress gateway pods, follow these steps: - Ensure that your worker nodes are either in a private subnet, or they are using Elastic IPs for their public IP. > **WARNING:** If your worker nodes are using ["standard" VPC public IPs](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-ip-addressing.html#vpc-public-ipv4-addresses), adding the Elastic IP to the node triggers the node to lose its VPC Public IP. This is because in AWS networking, a node is not allowed to have a VPC public IP and an Elastic IP. However, we have found this is only enforced at boot time. - Ensure that your VPC has an [Internet Gateway](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html) and a (default) route to the Internet Gateway from the AWS subnets used for egress gateways. (This is a standard requirement for Elastic IPs in AWS.) - Create one or more Elastic IPs for the deployment. This can be done through the AWS Console or using the AWS command line interface. - Add the Elastic IPs to the egress gateway resource. ```yaml aws: nativeIP: Enabled elasticIPs: ["37.1.2.3", "43.2.5.6"] ``` Once the update has rolled out, Calico Cloud will try to add one of the requested Elastic IPs to each pod in the deployment. ### Optionally enable ECMP load balancing[​](#optionally-enable-ecmp-load-balancing) If you are provisioning multiple egress gateways for a given client pod, and you want traffic from that client to load balance across the available gateways, set the `fib_multipath_hash_policy` [sysctl](https://sysctl-explorer.net/net/ipv4/fib_multipath_hash_policy/) to allow that: ```bash sudo sysctl -w net.ipv4.fib_multipath_hash_policy=1 ``` You will need this on each node with clients that you want to load balance across multiple egress gateways. ### Verify the feature operation[​](#verify-the-feature-operation) To verify the feature operation, cause the application pod to initiate a connection to a server outside the cluster, and observe -- for example using tcpdump -- the source IP of the connection packet as it reaches the server. > **SECONDARY:** In order for such a connection to complete, the server must know how to route back to the egress gateway's IP. By way of a concrete example, you could use netcat to run a test server outside the cluster (outside AWS if you're using Elastic IPs); for example: ```bash docker run --net=host --privileged subfuzion/netcat -v -l -k -p 8089 ``` Then provision an egress IP Pool, and egress gateways, as above. Then deploy a pod, with egress annotations as above, and with any image that includes netcat, for example: ```bash kubectl apply -f - < -n -- nc 8089 ` should be the IP address of the netcat server. Then, if you check the logs or output of the netcat server, you should see: ```text Connection from received ``` with `` being one of the IPs of the egress IP pool that you provisioned. ### Control the use of egress gateways[​](#control-the-use-of-egress-gateways) If a cluster ascribes special meaning to traffic flowing through egress gateways, it will be important to control when cluster users can configure their pods and namespaces to use them, so that non-special pods cannot impersonate the special meaning. If namespaces in a cluster can only be provisioned by cluster admins, one option is to enable egress gateway function only on a per-namespace basis. Then only cluster admins will be able to configure any egress gateway usage. Otherwise -- if namespace provisioning is open to users in general, or if it's desirable for egress gateway function to be enabled both per-namespace and per-pod -- a [Kubernetes admission controller](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/) will be needed. This is a task for each deployment to implement for itself, but possible approaches include the following. 1. Decide whether a given Namespace or Pod is permitted to use egress annotations at all, based on other details of the Namespace or Pod definition. 2. Evaluate egress annotation selectors to determine the egress gateways that they map to, and decide whether that usage is acceptable. 3. Impose the cluster's own bespoke scheme for a Namespace or Pod to identify the egress gateways that it wants to use, less general than Calico Cloud's egress annotations. Then the admission controller would police those bespoke annotations (that that cluster's users could place on Namespace or Pod resources) and either reject the operation in hand, or allow it through after adding the corresponding Calico Cloud egress annotations. ### Policy enforcement for flows via an egress gateway[​](#policy-enforcement-for-flows-via-an-egress-gateway) For an outbound connection from a client pod, via an egress gateway, to a destination outside the cluster, there is more than one possible enforcement point for policy: The path of the traffic through policy is as follows: 1. Packet leaves the client pod and passes through its egress policy. 2. The packet is encapsulated by the client pod's host and sent to the egress gateway 3. The encapsulated packet is sent from the host to the egress gateway pod. 4. The egress gateway pod de-encapsulates the packet and sends the packet out again with its own address. 5. The packet leaves the egress gateway pod through its egress policy. To ensure correct operation, (as of v3.15) the encapsulated traffic between host and egress gateway is auto-allowed by Calico Cloud and other ingress traffic is blocked. That means that there are effectively two places where policy can be applied: 1. on egress from the client pod 2. on egress from the egress gateway pod (see limitations below). The policy applied at (1) is the most powerful since it implicitly sees the original source of the traffic (by virtue of being attached to that original source). It also sees the external destination of the traffic. Since an egress gateway will never originate its own traffic, one option is to rely on policy applied at (1) and to allow all traffic to at (2) (either by applying no policy or by applying an "allow all"). Alternatively, for maximum "defense in depth" applying policy at both (1) and (2) provides extra protection should the policy at (1) be disabled or bypassed by an attacker. Policy at (2) has the following limitations: - [Domain-based policy](https://docs.tigera.io/calico-cloud/network-policy/domain-based-policy) is not supported at egress from egress gateways. It will either fail to match the expected traffic, or it will work intermittently if the egress gateway happens to be scheduled to the same node as its clients. This is because any DNS lookup happens at the client pod. By the time the policy reaches (2) the DNS information is lost and only the IP addresses of the traffic are available. - The traffic source will appear to be the egress gateway pod, the source information is lost in the address translation that occurs inside the egress gateway pod. That means that policies at (2) will usually take the form of rules that match only on destination port and IP address, either directly in the rule (via a CIDR match) or via a (non-domain based) NetworkSet. Matching on source has little utility since the IP will always be the egress gateway and the port of translated traffic is not always preserved. > **SECONDARY:** Since v3.15.0, Calico Cloud also sends health probes to the egress gateway pods from the nodes where their clients are located. In iptables mode, this traffic is auto-allowed at egress from the host and ingress to the egress gateway. In eBPF mode, the probe traffic can be blocked by policy, so you must ensure that this traffic is allowed; this should be fixed in an upcoming patch release. ## Upgrade egress gateways[​](#upgrade-egress-gateways) From v3.16, egress gateway deployments are managed by the Tigera Operator. - When upgrading from a pre-v3.16 release, no automatic upgrade will occur. To upgrade a pre-v3.16 egress gateway deployment, create an equivalent EgressGateway resource with the same namespace and the same name as mentioned [above](#deploy-a-group-of-egress-gateways); the operator will then take over management of the old Deployment resource, replacing it with the upgraded version. - Use `kubectl apply` to create the egress gateway resource. Tigera Operator will read the newly created resource and wait for the other Calico Cloud components to be upgraded. Once the other Calico Cloud components are upgraded, Tigera Operator will upgrade the existing egress gateway deployment with the new image. By default, upgrading egress gateways will sever any connections that are flowing through them. To minimise impact, the egress gateway feature supports some advanced options that give feedback to affected pods. For more details see the [egress gateway maintenance guide](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-maintenance). ## Additional resources[​](#additional-resources) Please see also: - The `egressIP...` and `aws...` fields of the [FelixConfiguration resource](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig#spec). - [Troubleshooting egress gateways](https://docs.tigera.io/calico-cloud/networking/egress/troubleshoot). - [Additional configuration for egress gateway maintenance](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-maintenance) ### Configure egress gateways, Azure ## Big picture[​](#big-picture) Control the source IP address seen by external services/appliances by routing the traffic from certain pods through egress gateways. Use native VNet subnet IP addresses for the egress gateways so that the IPs are valid in the Azure fabric. ## Value[​](#value) Controlling the source IP seen when traffic leaves the cluster allows groups of pods to be identified by external firewalls, appliances and services (even as the groups are scaled up/down or pods restarted). Calico Cloud controls the source IP by directing traffic through one or more "egress gateway" pods, which change the source IP of the traffic to their own IP. The egress gateways used can be chosen at the pod or namespace scope allowing for flexibility in how the cluster is seen from outside. In Azure, egress gateway source IP addresses are chosen from an arbitrary user-defined IP pool using Calico Cloud IPAM. egress gateway pods use dedicated IPPools to use as source IP addresses, which enables static configuration of external appliances. ## Concepts[​](#concepts) ### Egress gateway[​](#egress-gateway) An egress gateway acts as a transit pod for the outbound application traffic that is configured to use it. As traffic leaving the cluster passes through the egress gateway, its source IP is changed to that of the egress gateway pod, and the traffic is then forwarded on. ### Source IP[​](#source-ip) When an outbound application flow leaves the cluster, its IP packets will have a source IP. This begins as the pod IP of the pod that originated the flow, then: - *If no egress gateway is configured* and the pod IP came from an [IP pool](https://docs.tigera.io/calico-cloud/reference/resources/ippool) with `natOutgoing: true`, the node hosting the pod will change the source IP to its own as the traffic leaves the host. This allows the pod to communicate with external service even though the external network is unaware of the pod's IP. - *If the pod is configured with an egress gateway*, the traffic is first forwarded to the egress gateway, which changes the source IP to its own and then sends the traffic on. To function correctly, egress gateways should have IPs from an IP pool with `natOutgoing: false`, meaning their host forwards the packet onto the network without changing the source IP again. Since the egress gateway's IP is visible to the underlying network fabric, the fabric must be configured to know about the egress gateway's IP and to send response traffic back to the same host. ### Azure VNets and subnets[​](#azure-vnets-and-subnets) An Azure VNet is a virtual network that is, by default, logically isolated from other VNets. Each VNet has one or more (often large) CIDR blocks associated with it (for example `10.0.0.0/16`). In general, VNet CIDRs may overlap, but only if the VNets remain isolated. Azure allows VNets to be peered with each other through VNet Peerings. VNets can only be peered if *none* of their associated CIDRs overlap. Each VNet has one or more VNet subnets associated with it, each subnet owns a non-overlapping part of one of the VNet's CIDR blocks. VNet subnets span all availability zones in a region, so it is possible to distribute egress gateway resources across availability zones in a region without the need to repeat the configuration per-AZ. ### Azure VNet and ExpressRoute peerings[​](#azure-vnet-and-expressroute-peerings) Azure [VNet Peerings](https://learn.microsoft.com/en-us/azure/virtual-network/virtual-network-peering-overview) allow multiple VNets to be connected together. Similarly, [ExpressRoute](https://learn.microsoft.com/en-us/azure/expressroute/expressroute-introduction) allows external datacenters to be connected to an Azure VNet. Peered VPCs and datacenters communicate using private IPs as if they were all on one large private network. By advertising routes to Azure fabric using Azure Route Servers, egress gateways can be assigned private IPs allowing them to communicate without NAT within the same VPC, with peered VPCs, and, with peered datacenters. ### Azure Route Server[​](#azure-route-server) Azure [Route Server](https://learn.microsoft.com/en-us/azure/route-server/) is a managed networking service that allows a network virtual appliance, like Calico Cloud, to dynamically configure Azure fabric by exchanging routing information using Border Gateway Protocol (BGP). Calico Cloud can establish BGP sessions with an Azure Route Server in a VNet to advertise the IPs of egress gateways to that VNet. The learned routes are then propagated to the rest of VNets through VNet peering, or to external datacenters through ExpressRoute, allowing communication with egress gateway. ## Before you begin[​](#before-you-begin) **Required** - Calico CNI - Open port UDP 4790 on the host **Not Supported** - Azure VNet CNI Calico Cloud CNI and IPAM are required. The ability to control the egress gateway’s IP is a feature of Calico Cloud CNI and IPAM. Azure VNet CNI does not support that feature, so it is incompatible with egress gateways. ## How to[​](#how-to) - [Choose route reflectors](#choose-route-reflectors) - [Create Azure Route Server](#create-azure-route-server) - [Disable the default BGP node-to-node mesh](#disable-the-default-bgp-node-to-node-mesh) - [Enable BGP](#enable-bgp) - [Provision an egress IP pool](#provision-an-egress-ip-pool) - [(Optional) Limit number of route advertisement](#limit-number-of-route-advertisement) - [Configure route reflector](#configure-route-reflector) - [Enable egress gateway support](#enable-egress-gateway-support) - [Deploy a group of egress gateways](#deploy-a-group-of-egress-gateways) - [Configure iptables backend for egress gateways](#configure-iptables-backend-for-egress-gateways) - [Configure namespaces and pods to use egress gateways](#configure-namespaces-and-pods-to-use-egress-gateways) - [(Optional) Enable ECMP load balancing](#optionally-enable-ecmp-load-balancing) - [Verify the feature operation](#verify-the-feature-operation) - [Control the use of egress gateways](#control-the-use-of-egress-gateways) - [Policy enforcement for flows via an egress gateway](#policy-enforcement-for-flows-via-an-egress-gateway) ### Choose route reflectors[​](#choose-route-reflectors) It is possible to establish BGP connections between all Calico Cloud nodes and Azure Route Servers, but to avoid hitting Azure Route Server peers [limit](https://learn.microsoft.com/en-us/azure/route-server/overview#route-server-limits), it is better to select some nodes as route reflectors and set up BGP connections between those and Azure Route Server. The number of route reflectors depends on the cluster size, but it is recommended to have at least 3 at all times, and Azure Route Servers supports up to 8 peers in a VNet. ### Create Azure Route Server[​](#create-azure-route-server) Deploy Azure [Route Server](https://learn.microsoft.com/en-us/azure/route-server/) in the VNet (hub or spoke VNet) that routes egress addresses. Then, add the selected route reflectors as peers to the Azure Route Server. > **SECONDARY:** > > - The BGP connections between Calico route reflectors and Azure Route Servers are critical for the functionality of egress gateways. It is important to maintain route reflectors with care, and to make sure there are always enough healthy route reflectors. > - If possible, assign a static address to the route reflectors so after reboots the same address is kept. > - In AKS, it is [recommended](https://learn.microsoft.com/en-us/azure/aks/use-system-pools?tabs=azure-cli#system-and-user-node-pools) to run applications in user node pools and leave system node pools for running critical system pods. The nodes in system node pools are perfect Route Reflector candidates. ### Disable the default BGP node-to-node mesh[​](#disable-the-default-bgp-node-to-node-mesh) The default Calico Cloud **node-to-node BGP mesh** may be turned off to enable other BGP topologies. To do this, modify the default **BGP configuration** resource. Run the following command to disable the BGP full-mesh: ```bash kubectl apply -f - < **SECONDARY:** Disabling the node-to-node mesh will break pod networking until/unless you configure replacement BGP peerings using BGPPeer resources. You may configure the BGPPeer resources before disabling the node-to-node mesh to avoid pod networking breakage. ### Enable BGP[​](#enable-bgp) Usually, in the default installation of Calico Cloud in a public cloud, BGP is disabled as it is not used. However, running egress gateways in Azure relies on BGP connections between Calico Cloud and Azure Route Servers. To enable BGP routing protocol, set `bgp` field to `Enabled` in the default installation: ```bash kubectl patch installation default --type='merge' -p '{"spec": {"calicoNetwork": {"bgp": "Enabled"}}}' ``` and then restart the Calico Cloud daemonSet: ```bash kubectl rollout restart ds calico-node -n calico-system ``` ### Provision an egress IP pool[​](#provision-an-egress-ip-pool) Provision a small IP Pool with the range of source IPs that you want to use for a particular application when it connects to an external service. For example: ```bash kubectl apply -f - < projectcalico.org/RouteReflectorClusterID=244.0.0.1 ``` Typically, you will want to label this node to indicate that it is a route reflector, allowing it to be easily selected by a BGPPeer resource. You can do this with kubectl. For example: ```bash kubectl label node route-reflector=true ``` Now it is easy to configure route reflector nodes to peer with each other and other non-route-reflector nodes using label selectors. For example: ```yaml kind: BGPPeer apiVersion: projectcalico.org/v3 metadata: name: peer-with-route-reflectors spec: nodeSelector: all() peerSelector: route-reflector == 'true' ``` Finally, Add the IP addresses of Azure Route Servers, usually there are two, as peers to route reflectors. For example, if the Calico Cloud cluster is in a subnet with 10.224.0.0 as network address, the 10.224.0.1 is, [by definition](https://learn.microsoft.com/en-us/azure/virtual-network/virtual-networks-faq#are-there-any-restrictions-on-using-ip-addresses-within-these-subnets), the gateway and Azure Route Servers are deployed with ASN 65515 and IP addresses of 10.225.0.4 and 10.225.0.5, create two **BGPPeer** resources: ```yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: azure-route-server-a spec: peerIP: 10.225.0.4 reachableBy: 10.224.0.1 asNumber: 65515 keepOriginalNextHop: true nodeSelector: route-reflector == 'true' filters: - export-egress-ips --- apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: azure-route-server-b spec: peerIP: 10.225.0.5 reachableBy: 10.224.0.1 asNumber: 65515 keepOriginalNextHop: true nodeSelector: route-reflector == 'true' filters: - export-egress-ips ``` > **SECONDARY:** > > - Adding `routeReflectorClusterID` to a node spec will remove it from the node-to-node mesh immediately, tearing down the existing BGP sessions. Adding the BGP peering will bring up new BGP sessions. This will cause a short (about 2 seconds) disruption to data plane traffic of workloads running in the nodes where this happens. To avoid this, make sure no workloads are running on the nodes, by provisioning new nodes or by running `kubectl drain` on the node (which may itself cause a disruption as workloads are drained). > - It is important to set `keepOriginalNextHop: true` since route reflectors advertise routes on behalf of other nodes. Advertised routes to Azure Route Servers should have the original next hop otherwise the return packets will be sent to route reflectors, and get dropped. > - It is mandatory to set `reachableBy` field set to the gateway of the subnet Calico Cloud cluster is running in for peering with Azure Route Servers to prevent BGP connection flapping. See [BGP peer](https://docs.tigera.io/calico-cloud/reference/resources/bgppeer) for more information. > - Including `filters` to apply the BGP filter configured in the previous section, is optional. ### Enable egress gateway support[​](#enable-egress-gateway-support) In the default **FelixConfiguration**, set the `egressIPSupport` field to `EnabledPerNamespace` or `EnabledPerNamespaceOrPerPod`, according to the level of support that you need in your cluster. For support on a per-namespace basis only: ```bash kubectl patch felixconfiguration default --type='merge' -p \ '{"spec":{"egressIPSupport":"EnabledPerNamespace"}}' ``` Or for support both per-namespace and per-pod: ```bash kubectl patch felixconfiguration default --type='merge' -p \ '{"spec":{"egressIPSupport":"EnabledPerNamespaceOrPerPod"}}' ``` > **SECONDARY:** > > - `egressIPSupport` must be the same on all cluster nodes, so you should set them only in the `default` FelixConfiguration resource. > - The operator automatically enables the required policy sync API in the FelixConfiguration. ### Deploy a group of egress gateways[​](#deploy-a-group-of-egress-gateways) Use a Kubernetes Deployment to deploy a group of egress gateways. Using the example of the "red" egress gateway cluster, we use several features of Kubernetes and Calico Cloud in tandem to get a cluster of egress gateways that uses the user defined IP addresses: ```bash kubectl apply -f - < # - # timeoutSeconds: 15 # intervalSeconds: 5 # httpProbe: # urls: # - # - # timeoutSeconds: 30 # intervalSeconds: 10 template: metadata: labels: egress-code: red spec: terminationGracePeriodSeconds: 0 nodeSelector: kubernetes.io/os: linux topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: egress-code: red EOF ``` - `replicas: 2` tells Kubernetes to schedule two egress gateways in the "red" cluster. - This annotation tells Calico Cloud IPAM to use one of the "egress-ip-red-pool" IP pool. External services and appliances can recognise "red" traffic because it will all come from the CIDRs of the "red" IP pool. - The following [topology spread constraint](https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/) ensures that Kubernetes spreads the Egress gateways evenly between AZs (assuming that your nodes are labeled with the expected [well-known label](https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone) `topology.kubernetes.io/zone`): ```yaml topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: egress-code: red ``` - The labels are arbitrary. You can choose whatever names and values are convenient for your cluster's Namespaces and Pods to refer to in their egress selectors. If labels are not specified, a default label `projectcalico.org/egw`:`name` will be added by the Tigera Operator. - icmpProbe may be used to specify the Probe IPs, ICMP interval and timeout in seconds. `ips` if set, the egress gateway pod will probe each IP periodically using an ICMP ping. If all pings fail then the egress gateway will report non-ready via its health port. `intervalSeconds` controls the interval between probes. `timeoutSeconds` controls the timeout before reporting non-ready if no probes succeed. ```yaml icmpProbe: ips: - - timeoutSeconds: 20 intervalSeconds: 10 ``` - httpProbe may be used to specify the Probe URLs, HTTP interval and timeout in seconds. `urls` if set, the egress gateway pod will probe each external service periodically. If all probes fail then the egress gateway will report non-ready via its health port. `intervalSeconds` controls the interval between probes. `timeoutSeconds` controls the timeout before reporting non-ready if all probes are failing. ```yaml httpProbe: urls: - - timeoutSeconds: 30 intervalSeconds: 10 ``` - Please refer to the [operator reference docs](https://docs.tigera.io/calico-cloud/reference/installation/api) for details about the egress gateway resource type. > **SECONDARY:** > > - It is advisable to have more than one egress gateway per group, so that the egress IP function continues if one of the gateways crashes or needs to be restarted. When there are multiple gateways in a group, outbound traffic from the applications using that group is load-balanced across the available gateways. The number of `replicas` specified must be less than or equal to the number of free IP addresses in the IP Pool. > > - IPPool can be specified either by its name (e.g. `-name: egress-ip-red-pool`) or by its CIDR (e.g. `-cidr: 10.10.10.0/30`). > > - The labels are arbitrary. You can choose whatever names and values are convenient for your cluster's Namespaces and Pods to refer to in their egress selectors. > > The health port `8080` is used by: > > - The Kubernetes `readinessProbe` to expose the status of the egress gateway pod (and any ICMP/HTTP probes). > > - Remote pods to check if the egress gateway is "ready". Only "ready" egress gateways will be used for remote client traffic. This traffic is automatically allowed by Calico Cloud and no policy is required to allow it. Calico Cloud only sends probes to egress gateway pods that have a named "health" port. This ensures that during an upgrade, health probes are only sent to upgraded egress gateways. ### Configure iptables backend for egress gateways[​](#configure-iptables-backend-for-egress-gateways) The Tigera Operator configures egress gateways to use the same iptables backend as `calico-node`. To modify the iptables backend for egress gateways, you must change the `iptablesBackend` field in the [Felix configuration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig). ### Configure namespaces and pods to use egress gateways[​](#configure-namespaces-and-pods-to-use-egress-gateways) You can configure namespaces and pods to use an egress gateway by: - annotating the namespace or pod - applying an egress gateway policy to the namespace or pod. Using an egress gateway policy is more complicated, but it allows advanced use cases. #### Configure a namespace or pod to use an egress gateway (annotation method)[​](#configure-a-namespace-or-pod-to-use-an-egress-gateway-annotation-method) In a Calico Cloud deployment, the Kubernetes namespace and pod resources honor annotations that tell that namespace or pod to use particular egress gateways. These annotations are selectors, and their meaning is "the set of pods, anywhere in the cluster, that match those selectors". So, to configure all the pods in a namespace to use the egress gateways that are labelled with `egress-code: red`, you would annotate that namespace like this: ```bash kubectl annotate ns egress.projectcalico.org/selector="egress-code == 'red'" ``` By default, that selector can only match egress gateways in the same namespace. To select gateways in a different namespace, specify a `namespaceSelector` annotation as well, like this: ```bash kubectl annotate ns egress.projectcalico.org/namespaceSelector="projectcalico.org/name == 'default'" ``` Egress gateway annotations have the same [syntax and range of expressions](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy#selector) as the selector fields in Calico Cloud [network policy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy#entityrule). To configure a specific Kubernetes Pod to use egress gateways, specify the same annotations when creating the pod. For example: ```bash kubectl apply -f - < **SECONDARY:** CIDRs specified in rules in an egress gateway policy are matched in Longest Prefix Match(LPM) fashion. Calico Cloud rejects egress gateway policies that do any of the following: - The policy has no rule that specifies a gateway or a destination - The policy has a rule with empty `selector` or `namespaceSelector` fields. - The policy has two or more rules with the same destination. To configure all the pods in a namespace to use an egress gateway policy named `egw-policy1`, you could annotate the namespace like this: ```bash kubectl annotate ns egress.projectcalico.org/egressGatewayPolicy="egw-policy1" ``` To configure a specific Kubernetes pod to use the same policy, specify the same annotations when creating the pod. For example: ```bash kubectl apply -f - < **WARNING:** You must create the egress gateway policy before you apply it to a namespace or pod. If you attempt to apply an egress gateway policy that has not been created, Calico Cloud will block all traffic from the namespace or pod. ### (Optional) Enable ECMP load balancing[​](#optional-enable-ecmp-load-balancing) If you are provisioning multiple egress gateways for a given client pod, and you want traffic from that client to load balance across the available gateways, set the `fib_multipath_hash_policy` [sysctl](https://sysctl-explorer.net/net/ipv4/fib_multipath_hash_policy/) to allow that: ```bash sudo sysctl -w net.ipv4.fib_multipath_hash_policy=1 ``` You will need this on each node with clients that you want to load balance across multiple egress gateways. ### Verify the feature operation[​](#verify-the-feature-operation) To verify the feature operation, cause the application pod to initiate a connection to a server outside the cluster, and observe -- for example using tcpdump -- the source IP of the connection packet as it reaches the server. > **SECONDARY:** In order for such a connection to complete, the server must know how to route back to the egress gateway's IP. By way of a concrete example, you could use netcat to run a test server outside the cluster; for example: ```bash docker run --net=host --privileged subfuzion/netcat -v -l -k -p 8089 ``` Then provision an egress IP Pool, and egress gateways, as above. Then deploy a pod, with egress annotations as above, and with any image that includes netcat, for example: ```bash kubectl apply -f - < spec: containers: - name: alpine image: alpine command: ["/bin/sleep"] args: ["infinity"] EOF ``` Now you can use `kubectl exec` to initiate an outbound connection from that pod: ```bash kubectl exec -n -- nc 8089 ` should be the IP address of the netcat server. Then, if you check the logs or output of the netcat server, you should see: ```text Connection from received ``` with `` being one of the IPs of the egress IP pool that you provisioned. ### Control the use of egress gateways[​](#control-the-use-of-egress-gateways) If a cluster ascribes special meaning to traffic flowing through egress gateways, it will be important to control when cluster users can configure their pods and namespaces to use them, so that non-special pods cannot impersonate the special meaning. If namespaces in a cluster can only be provisioned by cluster admins, one option is to enable egress gateway function only on a per-namespace basis. Then only cluster admins will be able to configure any egress gateway usage. Otherwise -- if namespace provisioning is open to users in general, or if it's desirable for egress gateway function to be enabled both per-namespace and per-pod -- a [Kubernetes admission controller](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/) will be needed. This is a task for each deployment to implement for itself, but possible approaches include the following. 1. Decide whether a given Namespace or Pod is permitted to use egress annotations at all, based on other details of the Namespace or Pod definition. 2. Evaluate egress annotation selectors to determine the egress gateways that they map to, and decide whether that usage is acceptable. 3. Impose the cluster's own bespoke scheme for a Namespace or Pod to identify the egress gateways that it wants to use, less general than Calico Cloud's egress annotations. Then the admission controller would police those bespoke annotations (that that cluster's users could place on Namespace or Pod resources) and either reject the operation in hand, or allow it through after adding the corresponding Calico Cloud egress annotations. #### Policy enforcement for flows via an egress gateway[​](#policy-enforcement-for-flows-via-an-egress-gateway) For an outbound connection from a client pod, via an egress gateway, to a destination outside the cluster, there is more than one possible enforcement point for policy: The path of the traffic through policy is as follows: 1. Packet leaves the client pod and passes through its egress policy. 2. The packet is encapsulated by the client pod's host and sent to the egress gateway 3. The encapsulated packet is sent from the host to the egress gateway pod. 4. The egress gateway pod de-encapsulates the packet and sends the packet out again with its own address. 5. The packet leaves the egress gateway pod through its egress policy. To ensure correct operation, (as of v3.15) the encapsulated traffic between host and egress gateway is auto-allowed by Calico Cloud and other ingress traffic is blocked. That means that there are effectively two places where policy can be applied: 1. on egress from the client pod 2. on egress from the egress gateway pod (see limitations below). The policy applied at (1) is the most powerful since it implicitly sees the original source of the traffic (by virtue of being attached to that original source). It also sees the external destination of the traffic. Since an egress gateway will never originate its own traffic, one option is to rely on policy applied at (1) and to allow all traffic to at (2) (either by applying no policy or by applying an "allow all"). Alternatively, for maximum "defense in depth" applying policy at both (1) and (2) provides extra protection should the policy at (1) be disabled or bypassed by an attacker. Policy at (2) has the following limitations: - [Domain-based policy](https://docs.tigera.io/calico-cloud/network-policy/domain-based-policy) is not supported at egress from egress gateways. It will either fail to match the expected traffic, or it will work intermittently if the egress gateway happens to be scheduled to the same node as its clients. This is because any DNS lookup happens at the client pod. By the time the policy reaches (2) the DNS information is lost and only the IP addresses of the traffic are available. - The traffic source will appear to be the egress gateway pod, the source information is lost in the address translation that occurs inside the egress gateway pod. That means that policies at (2) will usually take the form of rules that match only on destination port and IP address, either directly in the rule (via a CIDR match) or via a (non-domain based) NetworkSet. Matching on source has little utility since the IP will always be the egress gateway and the port of translated traffic is not always preserved. > **SECONDARY:** Since v3.15.0, Calico Cloud also sends health probes to the egress gateway pods from the nodes where their clients are located. In iptables mode, this traffic is auto-allowed at egress from the host and ingress to the egress gateway. In eBPF mode, the probe traffic can be blocked by policy, so you must ensure that this traffic is allowed. ## Additional resources[​](#additional-resources) Please see also: - [Configure egress gateways, on-premises](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-on-prem) - [Troubleshooting egress gateways](https://docs.tigera.io/calico-cloud/networking/egress/troubleshoot). - [Additional configuration for egress gateway maintenance](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-maintenance) ### Optimize egress networking for workloads with long-lived TCP connections ## Big picture[​](#big-picture) React to egress gateway maintenance windows and minimize the impact of egress gateway downtime on sensitive workloads ## Value[​](#value) While most workloads benefit from the reduced downtime associated with increased replicas, there are some specific cases where increasing your number of egress gateways will not have as significant an effect on availability. One area where this applies is when your workloads must maintain long-lived TCP connections that are coupled to higher-level abstractions, such as database sessions, transactions, or similar. In these environments, if an egress gateway becomes unavailable, these long-lived sessions may close. Data may need to be retransmitted. And in some cases, data may be lost. ## Before you begin[​](#before-you-begin) These features require you to have configured a functioning egress gateway deployment in a cluster. For more information on deploying egress gateways, [see our other egress gateway guides](https://docs.tigera.io/calico-cloud/networking/egress/) ## How to[​](#how-to) - [Observe gateway maintenance impact](#observe-gateway-maintenance-impact) - [Expose gateway maintenance annotations to your application](#expose-gateway-maintenance-annotations-to-your-application) - [Reduce the impact of gateway downtime](#reduce-the-impact-of-gateway-downtime) ### Observe gateway maintenance impact[​](#observe-gateway-maintenance-impact) A number of egress-related annotations are automatically added to your workloads when an egress gateway they use is in the "terminating" phase. These annotations will outline *which* gateway is about to terminate, *when it began terminating*, and *when it will fully terminate*. This information can prove useful for conducting non-disruptive maintenance on your cluster, as it means any planned termination of egress pods will be communicated to dependent workloads. Before we can observe these annotations, we must first configure a *termination grace period* for our egress gateways. The termination grace period prolongs an egress gateway's termination phase, giving us a window to react to the termination. Without configuring the grace period, we would have a zero-second window to react to gateway termination. #### Add a termination grace period to egress gateway replicas[​](#add-a-termination-grace-period-to-egress-gateway-replicas) To widen our maintenance window, we can adjust the `terminationGracePeriodSeconds` field on our egress gateway pods. The amount of time we set for the termination grace period will dictate how much time a dependent workload has to prepare for a gateway going down. Let's add a termination grace period of 60 seconds to all pods in our egress gateway deployment, so that our egress-dependent workloads have a wider window to react: ```sh $ patch a termination grace period of 60s into egress gateway deployment $ kubectl get deploy NAME READY UP-TO-DATE AVAILABLE AGE egress-gateway 2/2 2 2 16m $ patch our deployment with a 60-second termination grace period $ kubectl patch deploy egress-gateway --type=merge -p '{"spec": {"template": {"spec": {"terminationGracePeriodSeconds": 60}}}}' deployment.apps/egress-gateway patched $ wait for change to rollout $ kubectl rollout status deploy/egress-gateway ``` > **SECONDARY:** > > - Making the above alterations to an egress gateway deployment will trigger a new rollout - you can monitor the rollout status with `kubectl rollout status deploy/` > - If your rollout seems to have stalled and egress gateway pods are stuck on "ContainerCreating" phase, it's possible the deployment's IPPool has been exhausted. You can check if this is the case by inspecting a stuck pod with `kubectl describe pod ` #### Inspect workload annotations[​](#inspect-workload-annotations) Once the updated egress gateway deployment rolls out, we're ready to observe the gateway maintenance annotations Calico Cloud adds to your dependent workloads. Let's simulate cluster maintenance by deleting an egress gateway pod. It should take 60 seconds to terminate - the amount of time defined by `terminationGracePeriodSeconds`. ```sh $ show pods in default namespace - two egress gateway pods and one application pod using them. $ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP application-with-long-lived-connections 1/1 Running 0 20m 192.168.192.210 egress-gateway-6576ccdf66-fxdvh 1/1 Running 0 3m 10.10.10.1 egress-gateway-6644fbb56b-5xbh2 1/1 Running 0 3m 10.10.10.0 $ delete one of the egress gateways being used by the application pod - do not block waiting for termination to finish $ kubectl delete pod egress-gateway-6576ccdf66-fxdvh --wait=false pod "egress-gateway-6576ccdf66-fxdvh" deleted ``` The gateway we just deleted should now wait in the "terminating" phase until its termination grace period expires, at-which point it will be deleted. If our application pod depends on the terminating egress gateway, we'll see gateway maintenance annotations added to the dependent application pod automatically, outlining what gateway is going down, when it began terminating, and when it will be deleted: ```sh $ observe the annotations added to the dependent application pod $ kubectl get pod application-with-long-lived-connections -o yaml apiVersion: v1 kind: Pod metadata: annotations: ... egress.projectcalico.org/gatewayMaintenanceFinishedTimestamp: "2022-04-19T16:00:18Z" egress.projectcalico.org/gatewayMaintenanceGatewayIP: 10.10.10.1 egress.projectcalico.org/gatewayMaintenanceStartedTimestamp: "2022-04-19T15:59:18Z" ... ``` Success! Our workload's annotations mark a 60-second maintenance window for the gateway we terminated, indicating when the egress gateway began terminating, and when it will fully terminate. > **SECONDARY:** > > - Adjusting egress deployments, say, by modifying the `terminationGracePeriodSeconds` field, will trigger a new rollout. > - Egress pods terminating due to a new rollout will behave the same as if they were deleted for maintenance - dependent workloads will receive gateway maintenance annotations, and gateway pods will terminate after their termination grace period has elapsed. > - Deleting an egress gateway in a way that overrides the termination grace period, say, by using `kubectl delete pod my-pod --grace-period=0`, will result in the gateway going down immediately, and dependent workloads will not have any time to react to the termination. ### Expose gateway maintenance annotations to your application[​](#expose-gateway-maintenance-annotations-to-your-application) While the presence of gateway maintenance annotations may be useful to a cluster administrator inspecting pods, it's not quite enough if our workload wishes to react to terminating egress gateways, say, by restarting its session gracefully before loss of connectivity. The [Kubernetes downward API](https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/#the-downward-api) provides a means of exposing pod information [as files](https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/#store-pod-fields) or as [environment variables](https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#use-pod-fields-as-values-for-environment-variables) within the pod. This value can then be polled by your workload, to react to changes as you see fit. Let's write a simple pod manifest that uses the downward API to expose maintenance annotations to the program running within: ```yaml apiVersion: v1 kind: Pod metadata: annotations: egress.projectcalico.org/selector: egress-code == 'red' egress.projectcalico.org/namespaceSelector: projectcalico.org/name == 'default' name: poll-my-own-annotations namespace: default spec: containers: - name: application-container image: k8s.gcr.io/busybox:1.24 command: ['sh', '-c'] args: - while true; do echo 'polling egress maintenance timestamp...'; if [[ -e /var/run/egress/gatewayMaintenanceStartedTimestamp ]]; then echo -n 'gatewayMaintenanceStartedTimestamp has value "'; cat /var/run/egress/gatewayMaintenanceStartedTimestamp; echo -en '"\n'; fi; sleep 3; done; volumeMounts: - name: egress-maintenance-started mountPath: /var/run/egress volumes: - name: egress-maintenance-started downwardAPI: items: - path: 'gatewayMaintenanceStartedTimestamp' fieldRef: fieldPath: metadata.annotations['egress.projectcalico.org/gatewayMaintenanceStartedTimestamp'] ``` This sample manifest will create a pod whose `gatewayMaintenanceStartedTimestamp` annotation is mounted to the file `/var/run/egress/gatewayMaintenanceStartedTimestamp`. The pod's main process is a script which polls the value of this file. After deleting an egress gateway this workload relies on, let's check its logs: ```sh $ kubectl logs poll-my-own-annotations polling egress maintenance timestamp... gatewayMaintenanceStartedTimestamp has value "" polling egress maintenance timestamp... gatewayMaintenanceStartedTimestamp has value "" polling egress maintenance timestamp... gatewayMaintenanceStartedTimestamp has value "" polling egress maintenance timestamp... gatewayMaintenanceStartedTimestamp has value "2022-04-19T17:24:46Z" polling egress maintenance timestamp... gatewayMaintenanceStartedTimestamp has value "2022-04-19T17:24:46Z" ``` We can see above that our script saw the value of the mounted volume change at the same time what we terminated our egress gateway pod. This work can be further developed to propagate notifications to our production workloads, without any need for polling kubernetes itself. **Note**: It's not recommended to couple your production applications to a Kubernetes client for the purposes of polling pod information, as it could give an attacker greater privileges if successful in compromising a workload. Instead, use a method such as the downward API that fully decouples the program. ### Reduce the impact of gateway downtime[​](#reduce-the-impact-of-gateway-downtime) So far we have observed egress gateway maintenance windows, added a termination grace period to gateway pods, and propagated maintenance information directly to our workloads. Finally, we are going to look at the `maxNextHops` annotation, which is designed to limit the impact of a terminating egress gateway. Below is a sample kubernetes deployment which was adapted from the earlier annotation-aware pod manifest. The deployment has 3 replicas, and is configured to use egress gateways: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: annotation-aware-workloads spec: replicas: 3 selector: matchLabels: app: annotation-aware-workload strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: app: annotation-aware-workload annotations: egress.projectcalico.org/selector: egress-code == 'red' egress.projectcalico.org/namespaceSelector: projectcalico.org/name == 'default' spec: containers: - name: application-container image: k8s.gcr.io/busybox:1.24 command: ['sh', '-c'] args: - while true; do echo "[${MY_POD_NAME}] polling egress maintenance timestamp..."; if [[ -e /var/run/egress/gatewayMaintenanceStartedTimestamp ]]; then echo -n "[${MY_POD_NAME}] gatewayMaintenanceStartedTimestamp has value '"; cat /var/run/egress/gatewayMaintenanceStartedTimestamp; echo -en "'\n"; fi; sleep 3; done; volumeMounts: - name: egress-maintenance-started mountPath: /var/run/egress env: - name: MY_POD_NAME valueFrom: fieldRef: fieldPath: metadata.name volumes: - name: egress-maintenance-started downwardAPI: items: - path: 'gatewayMaintenanceStartedTimestamp' fieldRef: fieldPath: metadata.annotations['egress.projectcalico.org/gatewayMaintenanceStartedTimestamp'] ``` #### Observing the impact of egress gateway maintenance[​](#observing-the-impact-of-egress-gateway-maintenance) We now want to conduct maintenance on a particular node in our cluster, resulting in one of our egress gateways terminating. In this scenario, since our dependent applications' connections are load-balanced evenly across all available egress gateways, all of our application pods will receive a maintenance window annotation. If we have configured our applications to react to such a window, we will see them all react at once in a thundering herd: ```sh $ begin the termination of an egress gateway pod $ kubectl delete pod egress-gateway-6576ccdf66-mtqzl --wait=false pod "egress-gateway-6576ccdf66-mtqzl" deleted $ lets collect logs from all pods in our maintenance-aware application deployment $ kubectl logs --selector=app=annotation-aware-workload [annotation-aware-workloads-7987f55c9f-f7mkq] polling egress maintenance timestamp... [annotation-aware-workloads-7987f55c9f-f7mkq] gatewayMaintenanceStartedTimestamp has value '' [annotation-aware-workloads-7987f55c9f-qtcs2] polling egress maintenance timestamp... [annotation-aware-workloads-7987f55c9f-qtcs2] gatewayMaintenanceStartedTimestamp has value '' [annotation-aware-workloads-7987f55c9f-z5x25] polling egress maintenance timestamp... [annotation-aware-workloads-7987f55c9f-z5x25] gatewayMaintenanceStartedTimestamp has value '' [annotation-aware-workloads-7987f55c9f-qtcs2] polling egress maintenance timestamp... [annotation-aware-workloads-7987f55c9f-qtcs2] gatewayMaintenanceStartedTimestamp has value '2022-04-20T12:24:34Z' [annotation-aware-workloads-7987f55c9f-f7mkq] polling egress maintenance timestamp... [annotation-aware-workloads-7987f55c9f-f7mkq] gatewayMaintenanceStartedTimestamp has value '2022-04-20T12:24:34Z' [annotation-aware-workloads-7987f55c9f-z5x25] polling egress maintenance timestamp... [annotation-aware-workloads-7987f55c9f-z5x25] gatewayMaintenanceStartedTimestamp has value '2022-04-20T12:24:34Z' ``` We can see in the above logs that all of our applications have been affected by the downtime of just a single egress gateway. In the worst case, this could lead to a window of downtime for the application, as all replicas scramble to restart their connections at once. To avoid this, lets use the `egress.projectcalico.org/maxNextHops` annotation to restrict the total number of gateways each application can depend on. #### Reducing the impact of egress gateway maintenance[​](#reducing-the-impact-of-egress-gateway-maintenance) To place a limit on the number of egress gateways an application can depend on, annotate the application's pod with the `egress.projectcalico.org/maxNextHops` annotation. Alternatively, to limit all pods in a certain namespace, annotate that namespace. Let's annotate all pods in our sample deployment from earlier: ```sh kubectl patch deploy annotation-aware-workloads --type=merge -p \ '{"spec": {"template": {"metadata": {"annotations": {"egress.projectcalico.org/maxNextHops": "1"} } } } }' ``` > **SECONDARY:** > > - Either a *pod or a namespace* can be annotated with `egress.projectcalico.org/maxNextHops`, however, the `egress.projectcalico.org/selector` annotation must also be present on the selected resource. > - If annotating pods, the `egressIPSupport` Felixconfiguration option must be set to `EnabledPerNamespaceOrPerPod`. > - If a pod's desired `maxNextHops` exceeds the total number of available egress gateways, scaling up the egress gateway deployment will result in the pod's egress networking updating until the desired number of gateways are being used. > - In all other cases, the `maxNextHops` annotation only takes effect at the time a pod is created. To ensure a pod's egress networking remains functional for its entire lifecycle, modifications to `maxNextHops` after a pod's creation will have no effect. For this reason, it's recommended that any egress gateway deployments have been scaled prior to deploying dependent workloads. After our patched sample deployment has been fully rolled out, each application pod should now depend on at most one egress gateway replica. Let's bring down another egress gateway pod and monitor our application logs: ```sh $ begin the termination of an egress gateway pod $ kubectl delete pod egress-gateway-6576ccdf66-c42v7 --wait=false pod "egress-gateway-6576ccdf66-c42v7" deleted $ collect logs from each application pod $ kubectl logs --selector=app=annotation-aware-workload [annotation-aware-workloads-565b6855b9-tjvqr] polling egress maintenance timestamp... [annotation-aware-workloads-565b6855b9-tjvqr] gatewayMaintenanceStartedTimestamp has value '' [annotation-aware-workloads-565b6855b9-s44pt] polling egress maintenance timestamp... [annotation-aware-workloads-565b6855b9-s44pt] gatewayMaintenanceStartedTimestamp has value '' [annotation-aware-workloads-565b6855b9-46cw5] polling egress maintenance timestamp... [annotation-aware-workloads-565b6855b9-46cw5] gatewayMaintenanceStartedTimestamp has value '' [annotation-aware-workloads-565b6855b9-tjvqr] polling egress maintenance timestamp... [annotation-aware-workloads-565b6855b9-tjvqr] gatewayMaintenanceStartedTimestamp has value '2022-04-20T12:53:32Z' [annotation-aware-workloads-565b6855b9-s44pt] polling egress maintenance timestamp... [annotation-aware-workloads-565b6855b9-s44pt] gatewayMaintenanceStartedTimestamp has value '' [annotation-aware-workloads-565b6855b9-46cw5] polling egress maintenance timestamp... [annotation-aware-workloads-565b6855b9-46cw5] gatewayMaintenanceStartedTimestamp has value '' ``` We can see from the above logs that only a single application pod has now been affected by the terminating egress gateway. The other pods have not received an annotation for a terminating gateway because they have chosen different gateways to depend on, and thus won't be affected. > **SECONDARY:** The subset of egress gateway replicas that each pod will depend on when using the `maxNextHops` annotation can't be manually selected. Calico Cloud selects a subset of replicas in such a way as to evenly distribute load across the whole replica set. ## Reference[​](#reference) **The following are annotations that Calico Cloud sets automatically on any egress-dependent pods:** | Annotation | Description | Datatype | Default value | Expected values | | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ---------- | ------------- | ---------------------------------------------------- | | `egress.projectcalico.org/gatewayMaintenanceGatewayIP` | Indicates the IP of a terminating egress gateway your pod is using. | IP Address | "" | Any IP within the egress gateway deployment's IPSet. | | `egress.projectcalico.org/gatewayMaintenanceStartedTimestamp` | Indicates when the egress gateway identified by `gatewayMaintenanceGatewayIP` began terminating. | String | "" | An RFC3339 date string | | `egress.projectcalico.org/gatewayMaintenanceFinishedTimestamp` | Indicates when the egress gateway identified by `gatewayMaintenanceGatewayIP` will finish terminating. | String | "" | An RFC3339 date string | **The following annotations are used to configure your egress-dependent workloads. These annotations can be set either on a namespace, or on a pod. If setting the annotations on pods, the `egressIPSupport` FelixConfiguration option must be set to `EnabledPerNamespaceOrPerPod`.** | Annotation | Description | Datatype | Default value | Possible values | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- | ------------- | ----------------------- | | `egress.projectcalico.org/maxNextHops` | Specifies the maximum number of egress gateway replicas from the selected deployment that a pod should depend on. Replicas will be chosen in a manner that attempts to balance load across the whole egress gateway replicaset. If unset, or set to "0", egress traffic will behave in the default manner (load balanced over all available gateways). | String | "" | "0", "1", "2", "3", ... | ## Additional resources[​](#additional-resources) - [Troubleshooting egress gateways](https://docs.tigera.io/calico-cloud/networking/egress/troubleshoot). ### Configure egress traffic to multiple external networks ## Big picture[​](#big-picture) Allows workloads from different namespaces of a Kubernetes cluster to egress onto different external networks that (may) have overlapping IPs with each other. ## Value[​](#value) When traffic from particular applications leaves the cluster to access an external destination, egress gateway enables users to control the source IP of that traffic. However, in this case, egress gateway sees all the external destinations as a flat L3 network. If a user has a network topology which involves multiple external networks and the service endpoints in those networks may have overlapping IPs, it becomes a mandatory requirement that Calico Cloud should be able to direct the egress traffic to the service endpoint in the correct network. For example, suppose a user has a network setup as below: There are two external networks (Red and Yellow) outside the cluster. A web server on network Red is exposing its service at `105.8.10.1`. A database server on network Yellow is exposing its service at `105.8.10.1` too. Then the cluster could have requirements as follows: Pods in Red namespace should send traffic via egress gateways to `105.8.10.1` on external network Red. Pods in Yellow namespace should send traffic via egress gateways to `105.8.10.1` on external network Yellow. External network support is introduced into Calico Cloud to meet those requirements. It allows user to associate a egress gateway to external networks so that the egress traffic originated from the client via that egress gateway will be routed to the service endpoints on the associated external networks. In a nutshell, the external network feature adds support for L3 segmentation of outgoing egress traffic. ## Prerequisites[​](#prerequisites) - IPs on multiple external networks may overlap with each other, however, those IPs must not overlap with the IPs of pods within the cluster or their hosts. - IPs of the BGP peers peering with the cluster should not overlap. ## Limitations[​](#limitations) - This feature should be used in combination with egress gateways. Sending traffic from a client pod to external networks without going through an egress gateway is not supported. - This feature only supports IPv4. ## Before you begin[​](#before-you-begin) This feature requires you to understand how to configure a functioning egress gateway deployment in a cluster. For more information on deploying egress gateways, [see our other egress gateway guides](https://docs.tigera.io/calico-cloud/networking/egress/). ## Concepts[​](#concepts) ### Egress node[​](#egress-node) A node in the cluster which is peering with external networks and populating routes to direct traffic from the egress gateway to the destinations on external networks. Any egress gateway which is associated to an external network should be scheduled to be running on a egress node. ### BGP bootstrap routes[​](#bgp-bootstrap-routes) BGP bootstrapping routes are routes on the node for traffic from that node to reach other nodes in the cluster, or other destinations outside the cluster, via whatever routers and external networks the user may need. This is considered outside of the scope of Calico Cloud, as both Calico Cloud and external BGP routers on external networks may require bootstrapping routes to be set up before peering with each other. ## How to[​](#how-to) ### Enable egress gateway support[​](#enable-egress-gateway-support) In the default **FelixConfiguration**, set the `egressIPSupport` field to `EnabledPerNamespace` or `EnabledPerNamespaceOrPerPod`, according to the level of granularity that you need in your cluster. For support on a per-namespace basis only: ```bash kubectl patch felixconfiguration default --type='merge' -p \ '{"spec":{"egressIPSupport":"EnabledPerNamespace"}}' ``` Or to support both per-namespace and per-pod: ```bash kubectl patch felixconfiguration default --type='merge' -p \ '{"spec":{"egressIPSupport":"EnabledPerNamespaceOrPerPod"}}' ``` ### Enable external network support[​](#enable-external-network-support) In the default **FelixConfiguration**, set the `externalNetworkSupport` field to `Enabled`. ```bash kubectl patch felixconfiguration default --type='merge' -p \ '{"spec":{"externalNetworkSupport":"Enabled"}}' ``` ### Create external network resources[​](#create-external-network-resources) Add an `ExternalNetwork` resource to the cluster for each external networks. For example, to create an `ExternalNetwork` for `red` network. ```bash kubectl create -f - < **WARNING:** Set `routeTableIndex` to the index of a linux kernel routing table that will be used for the routes associated with the external network. The value should be unique for each external network. The value must not be in the range of `RouteTableRanges` field in FelixConfiguration, as Calico Cloud uses tables in that range for various purposes. The kernel routing table index must not be used by other processes on the node. A conflict resulting in Calico Cloud using the same routing table index as a configured external network may result in network outages! Calico Cloud automatically picks up bootstrapping routes into the routing table used by an external network if those routes are pre-configured as a directly connected L2 route by the user in the main routing table. If the main routing table does not have the bootstrapping routes, the user should pre-configure this routing table with bootstrapping routes on each egress node outside of Calico Cloud to include whatever routes are needed for Calico Cloud to reach the BGP peer. ### Label egress node[​](#label-egress-node) Add a label, e.g. `egress: true`, to select multiple nodes in the cluster and mark them as egress nodes. - This label is used to satisfy the `nodeSelector` of the BGPPeer representing a BGP peer belongs to an external network. - This label is also used to enforce egress gateway pods associated external networks to be running on egress nodes with the help of setting `nodeSelector` field in pod spec. ### Create BGP peers[​](#create-bgp-peers) For each external router on an external network, create a BGPPeer resource that peers between the egress nodes and the external router; specify the name of the network in the `ExternalNetwork` field of the BGPPeer resource; set `nodeSelector` to the label which selects egress nodes. ```bash kubectl create -f - < # - # timeoutSeconds: 15 # intervalSeconds: 5 # httpProbe: # urls: # - # - # timeoutSeconds: 30 # intervalSeconds: 10 template: metadata: labels: egress-code: red spec: nodeSelector: kubernetes.io/os: linux terminationGracePeriodSeconds: 0 EOF ``` Where: - Where externalNetworks associates this egress gateway deployments with external network `red`. All traffic leaving the egress gateway to an external destination will be routed to the destination on external network `red`. - Node selector `egress: true` forces egress gateway pods of the deployment to be scheduled on egress nodes. > **WARNING:** If multiple networks in the `externalNetworkNames` list (of the egress gateway) advertise the same prefix to Calico Cloud then traffic to those prefixes will flow to a non-deterministic network, which may result in hard-to-debug connectivity issues. It is the user's responsibility to make sure it will never happen. ### Use egress gateways[​](#use-egress-gateways) Add annotations to namespaces or pods to configure them to use the relevant egress gateways created by the step above. Egress traffic from the application pods will be routed to destinations within the designated external networks. ## Additional resources[​](#additional-resources) - [Troubleshooting egress gateways](https://docs.tigera.io/calico-cloud/networking/egress/troubleshoot). ### Troubleshoot egress gateways - [Checklist of common problems](#checklist-of-common-problems) - [Connection to an egress gateway cannot be established](#connection-to-an-egress-gateway-cannot-be-established) - [Connection to an egress gateway is established, but destination is not getting correct IP](#connection-to-an-egress-gateway-is-established-but-destination-is-not-getting-correct-ip) ## Checklist of common problems[​](#checklist-of-common-problems) Use the following checklist to troubleshoot, or to collect details before opening a Support ticket. ### Is the egress gateway feature enabled?[​](#is-the-egress-gateway-feature-enabled) Egress gateway is disabled by default.  Have you enabled it in [Felix configuration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig) by setting `egressIPSupport` to `EnabledPerNamespace` or `EnabledPerNamespaceOrPerPod`? ### Does your egress gateway routing go through a router? (On-prem only.)[​](#does-your-egress-gateway-routing-go-through-a-router-on-prem-only) As shown in the following diagram, from the gateway to the destination, the source IP is the egress IP.  On the return path, from the destination back to the gateway, the destination IP is the egress IP. If there are any routers between the gateway and the destination, they must all know how to route the egress IP back to the gateway. If they don’t, the attempted connection cannot be established. ![egress-basic-routing](data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnhsaW5rPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rIiB3aWR0aD0iNjExIiBoZWlnaHQ9IjEyOCIgc3R5bGU9ImJhY2tncm91bmQtY29sb3I6I2ZmZiIgdmlld0JveD0iLTAuNSAtMC41IDYxMSAxMjgiPjxyZWN0IHdpZHRoPSIyNDAiIGhlaWdodD0iMTE4IiB5PSI5IiBmaWxsPSIjOGViYmZjIiBmaWxsLW9wYWNpdHk9Ii4zIiBzdHJva2U9IiNkZGQiIHN0cm9rZS1vcGFjaXR5PSIuMyIgcG9pbnRlci1ldmVudHM9Im5vbmUiIHJ4PSIxIiByeT0iMSIvPjxwYXRoIGZpbGw9IiMwNmMiIGQ9Ik04NC4wNCA0NkM3NS45IDQ2IDY5IDM5LjY2IDY5IDMwLjkgNjkgMjIuNDIgNzYuMDcgMTYgODQuMDEgMTYgOTIuMjUgMTYgOTkgMjMuMTkgOTkgMzAuOTMgOTkgMzkuMDkgOTIuMzYgNDYgODQuMDQgNDYiIHBvaW50ZXItZXZlbnRzPSJub25lIi8+PHBhdGggZmlsbD0iI2ZmZiIgZD0iTTc4LjYgMjYuOTFjLS44IDAtMS4zMi0uNjItMS4zMi0xLjM4IDAtLjY0LjUyLTEuMyAxLjMyLTEuMy43NiAwIDEuMzYuNjEgMS4zNiAxLjMgMCAuNzItLjUyIDEuMzgtMS4zNiAxLjM4bTEwLjc0IDBjLS44IDAtMS4zMi0uNjItMS4zMi0xLjM4IDAtLjY0LjUyLTEuMyAxLjMyLTEuMy43NiAwIDEuMzYuNjEgMS4zNiAxLjMgMCAuNzItLjUyIDEuMzgtMS4zNiAxLjM4bTAgMTAuNzhjLS44IDAtMS4zMi0uNjItMS4zMi0xLjM4IDAtLjY0LjUyLTEuMyAxLjMyLTEuMy43NiAwIDEuMzYuNjEgMS4zNiAxLjMgMCAuNzItLjUyIDEuMzgtMS4zNiAxLjM4bS0xMi4wNi0zLjkyYy0xLjU0IDAtMi44LTEuMjEtMi44LTIuNzkgMC0xLjIxIDEtMi43OCAyLjgtMi43OCAxLjU4IDAgMi43OCAxLjI1IDIuNzggMi43OGEyLjc3IDIuNzcgMCAwIDEtMi43OCAyLjc5bTYuNzQtNi43NWMtMS41NSAwLTIuODEtMS4yMS0yLjgxLTIuNzkgMC0xLjIyIDEuMDEtMi43OCAyLjgxLTIuNzggMS41NyAwIDIuNzggMS4yNSAyLjc4IDIuNzhhMi43NyAyLjc3IDAgMCAxLTIuNzggMi43OW02LjcxIDYuNzVjLTEuNTUgMC0yLjgxLTEuMjEtMi44MS0yLjc5IDAtMS4yMSAxLjAxLTIuNzggMi44MS0yLjc4IDEuNTcgMCAyLjc4IDEuMjUgMi43OCAyLjc4YTIuNzcgMi43NyAwIDAgMS0yLjc4IDIuNzltLTYuNzQtLjY5Yy0xLjEzIDAtMi4wOS0uODctMi4wOS0yLjEgMC0xLjA2LjgzLTIuMDkgMi4xLTIuMDkgMS4wOSAwIDIuMTEuODQgMi4xMSAyLjEgMCAxLjIxLS45NSAyLjA5LTIuMTIgMi4wOW0tNS4zOSA0LjYxYy0uOCAwLTEuMzItLjYyLTEuMzItMS4zOCAwLS42NC41Mi0xLjMgMS4zMi0xLjMuNzYgMCAxLjM2LjYxIDEuMzYgMS4zIDAgLjcyLS41MiAxLjM4LTEuMzYgMS4zOG01LjQyIDIuODZjLTEuNTUgMC0yLjgxLTEuMjEtMi44MS0yLjc5IDAtMS4yMSAxLjAxLTIuNzggMi44MS0yLjc4IDEuNTcgMCAyLjc4IDEuMjUgMi43OCAyLjc4YTIuNzcgMi43NyAwIDAgMS0yLjc4IDIuNzkiIHBvaW50ZXItZXZlbnRzPSJub25lIi8+PHJlY3Qgd2lkdGg9IjkwIiBoZWlnaHQ9IjQ1IiB4PSIxMCIgeT0iNjIiIGZpbGw9IiNmZmYiIHBvaW50ZXItZXZlbnRzPSJub25lIiByeD0iNi43NSIgcnk9IjYuNzUiLz48c3dpdGNoIHRyYW5zZm9ybT0idHJhbnNsYXRlKC0uNSAtLjUpIj48Zm9yZWlnbk9iamVjdCB3aWR0aD0iMTAwJSIgaGVpZ2h0PSIxMDAlIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIgcmVxdWlyZWRGZWF0dXJlcz0iaHR0cDovL3d3dy53My5vcmcvVFIvU1ZHMTEvZmVhdHVyZSNFeHRlbnNpYmlsaXR5IiBzdHlsZT0ib3ZlcmZsb3c6dmlzaWJsZTt0ZXh0LWFsaWduOmxlZnQiPjxkaXYgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkveGh0bWwiIHN0eWxlPSJkaXNwbGF5OmZsZXg7YWxpZ24taXRlbXM6dW5zYWZlIGNlbnRlcjtqdXN0aWZ5LWNvbnRlbnQ6dW5zYWZlIGNlbnRlcjt3aWR0aDo4OHB4O2hlaWdodDoxcHg7cGFkZGluZy10b3A6ODVweDttYXJnaW4tbGVmdDoxMXB4Ij48ZGl2IHN0eWxlPSJib3gtc2l6aW5nOmJvcmRlci1ib3g7Zm9udC1zaXplOjA7dGV4dC1hbGlnbjpjZW50ZXIiPjxkaXYgc3R5bGU9ImRpc3BsYXk6aW5saW5lLWJsb2NrO2ZvbnQtc2l6ZToxMnB4O2ZvbnQtZmFtaWx5OlZlcmRhbmE7Y29sb3I6IzAwMDtsaW5lLWhlaWdodDoxLjI7cG9pbnRlci1ldmVudHM6bm9uZTtmb250LXdlaWdodDo3MDA7d2hpdGUtc3BhY2U6bm9ybWFsO292ZXJmbG93LXdyYXA6bm9ybWFsIj48Zm9udCBzdHlsZT0iZm9udC1zaXplOjExcHg7Zm9udC13ZWlnaHQ6NDAwIj5DbGllbnQ8L2ZvbnQ+PC9kaXY+PC9kaXY+PC9kaXY+PC9mb3JlaWduT2JqZWN0Pjx0ZXh0IHg9IjU1IiB5PSI4OCIgZm9udC1mYW1pbHk9IlZlcmRhbmEiIGZvbnQtc2l6ZT0iMTIiIGZvbnQtd2VpZ2h0PSJib2xkIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5DbGllbnQ8L3RleHQ+PC9zd2l0Y2g+PHJlY3Qgd2lkdGg9IjkwIiBoZWlnaHQ9IjQ1IiB4PSIxNDAiIHk9IjYyIiBmaWxsPSIjZmZmIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIgcng9IjYuNzUiIHJ5PSI2Ljc1Ii8+PHN3aXRjaCB0cmFuc2Zvcm09InRyYW5zbGF0ZSgtLjUgLS41KSI+PGZvcmVpZ25PYmplY3Qgd2lkdGg9IjEwMCUiIGhlaWdodD0iMTAwJSIgcG9pbnRlci1ldmVudHM9Im5vbmUiIHJlcXVpcmVkRmVhdHVyZXM9Imh0dHA6Ly93d3cudzMub3JnL1RSL1NWRzExL2ZlYXR1cmUjRXh0ZW5zaWJpbGl0eSIgc3R5bGU9Im92ZXJmbG93OnZpc2libGU7dGV4dC1hbGlnbjpsZWZ0Ij48ZGl2IHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hodG1sIiBzdHlsZT0iZGlzcGxheTpmbGV4O2FsaWduLWl0ZW1zOnVuc2FmZSBjZW50ZXI7anVzdGlmeS1jb250ZW50OnVuc2FmZSBjZW50ZXI7d2lkdGg6ODhweDtoZWlnaHQ6MXB4O3BhZGRpbmctdG9wOjg1cHg7bWFyZ2luLWxlZnQ6MTQxcHgiPjxkaXYgc3R5bGU9ImJveC1zaXppbmc6Ym9yZGVyLWJveDtmb250LXNpemU6MDt0ZXh0LWFsaWduOmNlbnRlciI+PGRpdiBzdHlsZT0iZGlzcGxheTppbmxpbmUtYmxvY2s7Zm9udC1zaXplOjEycHg7Zm9udC1mYW1pbHk6VmVyZGFuYTtjb2xvcjojMDAwO2xpbmUtaGVpZ2h0OjEuMjtwb2ludGVyLWV2ZW50czpub25lO2ZvbnQtd2VpZ2h0OjcwMDt3aGl0ZS1zcGFjZTpub3JtYWw7b3ZlcmZsb3ctd3JhcDpub3JtYWwiPjxmb250IHN0eWxlPSJmb250LXNpemU6MTFweDtmb250LXdlaWdodDo0MDAiPkdhdGV3YXk8L2ZvbnQ+PC9kaXY+PC9kaXY+PC9kaXY+PC9mb3JlaWduT2JqZWN0Pjx0ZXh0IHg9IjE4NSIgeT0iODgiIGZvbnQtZmFtaWx5PSJWZXJkYW5hIiBmb250LXNpemU9IjEyIiBmb250LXdlaWdodD0iYm9sZCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+R2F0ZXdheTwvdGV4dD48L3N3aXRjaD48cmVjdCB3aWR0aD0iMTIwIiBoZWlnaHQ9IjgwIiB4PSI0OTAiIHk9IjM3IiBmaWxsPSIjZmZmIiBzdHJva2U9IiM3ZWE2ZTAiIHBvaW50ZXItZXZlbnRzPSJub25lIiByeD0iMTIiIHJ5PSIxMiIvPjxzd2l0Y2ggdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLS41IC0uNSkiPjxmb3JlaWduT2JqZWN0IHdpZHRoPSIxMDAlIiBoZWlnaHQ9IjEwMCUiIHBvaW50ZXItZXZlbnRzPSJub25lIiByZXF1aXJlZEZlYXR1cmVzPSJodHRwOi8vd3d3LnczLm9yZy9UUi9TVkcxMS9mZWF0dXJlI0V4dGVuc2liaWxpdHkiIHN0eWxlPSJvdmVyZmxvdzp2aXNpYmxlO3RleHQtYWxpZ246bGVmdCI+PGRpdiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94aHRtbCIgc3R5bGU9ImRpc3BsYXk6ZmxleDthbGlnbi1pdGVtczp1bnNhZmUgY2VudGVyO2p1c3RpZnktY29udGVudDp1bnNhZmUgY2VudGVyO3dpZHRoOjExOHB4O2hlaWdodDoxcHg7cGFkZGluZy10b3A6NzdweDttYXJnaW4tbGVmdDo0OTFweCI+PGRpdiBzdHlsZT0iYm94LXNpemluZzpib3JkZXItYm94O2ZvbnQtc2l6ZTowO3RleHQtYWxpZ246Y2VudGVyIj48ZGl2IHN0eWxlPSJkaXNwbGF5OmlubGluZS1ibG9jaztmb250LXNpemU6MTJweDtmb250LWZhbWlseTpWZXJkYW5hO2NvbG9yOiMwMDA7bGluZS1oZWlnaHQ6MS4yO3BvaW50ZXItZXZlbnRzOm5vbmU7Zm9udC13ZWlnaHQ6NzAwO3doaXRlLXNwYWNlOm5vcm1hbDtvdmVyZmxvdy13cmFwOm5vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtd2VpZ2h0OjQwMCI+RGVzdGluYXRpb248L3NwYW4+PC9kaXY+PC9kaXY+PC9kaXY+PC9mb3JlaWduT2JqZWN0Pjx0ZXh0IHg9IjU1MCIgeT0iODEiIGZvbnQtZmFtaWx5PSJWZXJkYW5hIiBmb250LXNpemU9IjEyIiBmb250LXdlaWdodD0iYm9sZCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+RGVzdGluYXRpb248L3RleHQ+PC9zd2l0Y2g+PHN3aXRjaCB0cmFuc2Zvcm09InRyYW5zbGF0ZSgtLjUgLS41KSI+PGZvcmVpZ25PYmplY3Qgd2lkdGg9IjEwMCUiIGhlaWdodD0iMTAwJSIgcG9pbnRlci1ldmVudHM9Im5vbmUiIHJlcXVpcmVkRmVhdHVyZXM9Imh0dHA6Ly93d3cudzMub3JnL1RSL1NWRzExL2ZlYXR1cmUjRXh0ZW5zaWJpbGl0eSIgc3R5bGU9Im92ZXJmbG93OnZpc2libGU7dGV4dC1hbGlnbjpsZWZ0Ij48ZGl2IHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hodG1sIiBzdHlsZT0iZGlzcGxheTpmbGV4O2FsaWduLWl0ZW1zOnVuc2FmZSBjZW50ZXI7anVzdGlmeS1jb250ZW50OnVuc2FmZSBjZW50ZXI7d2lkdGg6NThweDtoZWlnaHQ6MXB4O3BhZGRpbmctdG9wOjMycHg7bWFyZ2luLWxlZnQ6MTAwcHgiPjxkaXYgc3R5bGU9ImJveC1zaXppbmc6Ym9yZGVyLWJveDtmb250LXNpemU6MDt0ZXh0LWFsaWduOmNlbnRlciI+PGRpdiBzdHlsZT0iZGlzcGxheTppbmxpbmUtYmxvY2s7Zm9udC1zaXplOjEycHg7Zm9udC1mYW1pbHk6VmVyZGFuYTtjb2xvcjojMDAwO2xpbmUtaGVpZ2h0OjEuMjtwb2ludGVyLWV2ZW50czpub25lO2ZvbnQtd2VpZ2h0OjcwMDt3aGl0ZS1zcGFjZTpub3JtYWw7b3ZlcmZsb3ctd3JhcDpub3JtYWwiPkNsdXN0ZXI8L2Rpdj48L2Rpdj48L2Rpdj48L2ZvcmVpZ25PYmplY3Q+PHRleHQgeD0iMTI5IiB5PSIzNiIgZm9udC1mYW1pbHk9IlZlcmRhbmEiIGZvbnQtc2l6ZT0iMTIiIGZvbnQtd2VpZ2h0PSJib2xkIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5DbHVzdGVyPC90ZXh0Pjwvc3dpdGNoPjxwYXRoIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzAwMCIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBkPSJNMTA2LjM3IDg0aDI3LjI2IiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxwYXRoIHN0cm9rZT0iIzAwMCIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBkPSJtMTAxLjEyIDg0IDctMy41LTEuNzUgMy41IDEuNzUgMy41Wk0xMzguODggODRsLTcgMy41IDEuNzUtMy41LTEuNzUtMy41WiIgcG9pbnRlci1ldmVudHM9Im5vbmUiLz48cGF0aCBmaWxsPSIjN2VhNmUwIiBkPSJNMzY0Ljc2IDExN0MzNDMuNTUgMTE3IDMyOSAxMDEgMzI5IDg0LjQydi0zLjcxYzAtMTYuMTcgMTQuMDctMzIuNjggMzUuNTEtMzIuNjggMTguOTYgMCAzNS40NyAxNC4wMyAzNS40NyAzMy4wNGwuMDIgMi44OWMwIDE4Ljc4LTE2LjQ0IDMzLjA0LTM1LjI0IDMzLjA0IiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxwYXRoIGZpbGwtb3BhY2l0eT0iLjMiIGQ9Ik0zNjQuNjYgMTE0LjE3YzE5LjAxIDAgMzUuMzItMTQuMzcgMzUuMzItMzMuMWwuMDIgMi44OWMwIDE4LjgtMTYuNDUgMzMuMDQtMzUuMjQgMzMuMDQtMjEuMjggMC0zNS43Ni0xNS45OC0zNS43Ni0zMi41OHYtMy43MWMwIDE5Ljk3IDE3LjMxIDMzLjQ2IDM1LjY2IDMzLjQ2IiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxwYXRoIGZpbGw9IiNmZmYiIGQ9Im0zNjQuNDkgMTEwLjkxLTcuMjQtMTEuNzVoNC4zMlY4NC4wM2gtMTYuMjJ2NC4wMmwtMTIuNjEtNi43MyAxMi42MS02Ljg2djQuMTNoMTYuMjJWNjMuMzZoLTQuMzJsNy4yNC0xMS43NCA3LjI0IDExLjc0aC00LjMxdjE1LjIzaDE2LjIydi00LjEzbDEyLjYyIDYuODYtMTIuNjIgNi43NHYtNC4wM2gtMTYuMjJ2MTUuMTNoNC4zMVoiIHBvaW50ZXItZXZlbnRzPSJub25lIi8+PHN3aXRjaCB0cmFuc2Zvcm09InRyYW5zbGF0ZSgtLjUgLS41KSI+PGZvcmVpZ25PYmplY3Qgd2lkdGg9IjEwMCUiIGhlaWdodD0iMTAwJSIgcG9pbnRlci1ldmVudHM9Im5vbmUiIHJlcXVpcmVkRmVhdHVyZXM9Imh0dHA6Ly93d3cudzMub3JnL1RSL1NWRzExL2ZlYXR1cmUjRXh0ZW5zaWJpbGl0eSIgc3R5bGU9Im92ZXJmbG93OnZpc2libGU7dGV4dC1hbGlnbjpsZWZ0Ij48ZGl2IHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hodG1sIiBzdHlsZT0iZGlzcGxheTpmbGV4O2FsaWduLWl0ZW1zOnVuc2FmZSBjZW50ZXI7anVzdGlmeS1jb250ZW50OnVuc2FmZSBjZW50ZXI7d2lkdGg6NjlweDtoZWlnaHQ6MXB4O3BhZGRpbmctdG9wOjM0cHg7bWFyZ2luLWxlZnQ6MzMwcHgiPjxkaXYgc3R5bGU9ImJveC1zaXppbmc6Ym9yZGVyLWJveDtmb250LXNpemU6MDt0ZXh0LWFsaWduOmNlbnRlciI+PGRpdiBzdHlsZT0iZGlzcGxheTppbmxpbmUtYmxvY2s7Zm9udC1zaXplOjEycHg7Zm9udC1mYW1pbHk6VmVyZGFuYTtjb2xvcjojMDAwO2xpbmUtaGVpZ2h0OjEuMjtwb2ludGVyLWV2ZW50czpub25lO2ZvbnQtd2VpZ2h0OjcwMDt3aGl0ZS1zcGFjZTpub3JtYWw7b3ZlcmZsb3ctd3JhcDpub3JtYWwiPlJvdXRlcnM8L2Rpdj48L2Rpdj48L2Rpdj48L2ZvcmVpZ25PYmplY3Q+PHRleHQgeD0iMzY1IiB5PSIzOCIgZm9udC1mYW1pbHk9IlZlcmRhbmEiIGZvbnQtc2l6ZT0iMTIiIGZvbnQtd2VpZ2h0PSJib2xkIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5Sb3V0ZXJzPC90ZXh0Pjwvc3dpdGNoPjxwYXRoIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzAwMCIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBkPSJNMjMwIDc2LjVoMTAwIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxzd2l0Y2ggdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLS41IC0uNSkiPjxmb3JlaWduT2JqZWN0IHdpZHRoPSIxMDAlIiBoZWlnaHQ9IjEwMCUiIHBvaW50ZXItZXZlbnRzPSJub25lIiByZXF1aXJlZEZlYXR1cmVzPSJodHRwOi8vd3d3LnczLm9yZy9UUi9TVkcxMS9mZWF0dXJlI0V4dGVuc2liaWxpdHkiIHN0eWxlPSJvdmVyZmxvdzp2aXNpYmxlO3RleHQtYWxpZ246bGVmdCI+PGRpdiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94aHRtbCIgc3R5bGU9ImRpc3BsYXk6ZmxleDthbGlnbi1pdGVtczp1bnNhZmUgY2VudGVyO2p1c3RpZnktY29udGVudDp1bnNhZmUgY2VudGVyO3dpZHRoOjU4cHg7aGVpZ2h0OjFweDtwYWRkaW5nLXRvcDo2MnB4O21hcmdpbi1sZWZ0OjI2MXB4Ij48ZGl2IHN0eWxlPSJib3gtc2l6aW5nOmJvcmRlci1ib3g7Zm9udC1zaXplOjA7dGV4dC1hbGlnbjpjZW50ZXIiPjxkaXYgc3R5bGU9ImRpc3BsYXk6aW5saW5lLWJsb2NrO2ZvbnQtc2l6ZToxMXB4O2ZvbnQtZmFtaWx5OlZlcmRhbmE7Y29sb3I6IzAwMDtsaW5lLWhlaWdodDoxLjI7cG9pbnRlci1ldmVudHM6bm9uZTt3aGl0ZS1zcGFjZTpub3JtYWw7b3ZlcmZsb3ctd3JhcDpub3JtYWwiPlNvdXJjZTogRWdyZXNzIElQPC9kaXY+PC9kaXY+PC9kaXY+PC9mb3JlaWduT2JqZWN0Pjx0ZXh0IHg9IjI5MCIgeT0iNjUiIGZvbnQtZmFtaWx5PSJWZXJkYW5hIiBmb250LXNpemU9IjExIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5Tb3VyY2U6IEVnci4uLjwvdGV4dD48L3N3aXRjaD48cGF0aCBmaWxsPSJub25lIiBzdHJva2U9IiMwMDAiIHN0cm9rZS1taXRlcmxpbWl0PSIxMCIgZD0iTTQwMCA3Ni41aDgzLjYzIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxwYXRoIHN0cm9rZT0iIzAwMCIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBkPSJtNDg4Ljg4IDc2LjUtNyAzLjUgMS43NS0zLjUtMS43NS0zLjVaIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxzd2l0Y2ggdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLS41IC0uNSkiPjxmb3JlaWduT2JqZWN0IHdpZHRoPSIxMDAlIiBoZWlnaHQ9IjEwMCUiIHBvaW50ZXItZXZlbnRzPSJub25lIiByZXF1aXJlZEZlYXR1cmVzPSJodHRwOi8vd3d3LnczLm9yZy9UUi9TVkcxMS9mZWF0dXJlI0V4dGVuc2liaWxpdHkiIHN0eWxlPSJvdmVyZmxvdzp2aXNpYmxlO3RleHQtYWxpZ246bGVmdCI+PGRpdiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94aHRtbCIgc3R5bGU9ImRpc3BsYXk6ZmxleDthbGlnbi1pdGVtczp1bnNhZmUgY2VudGVyO2p1c3RpZnktY29udGVudDp1bnNhZmUgY2VudGVyO3dpZHRoOjU4cHg7aGVpZ2h0OjFweDtwYWRkaW5nLXRvcDoxMTJweDttYXJnaW4tbGVmdDo0MTFweCI+PGRpdiBzdHlsZT0iYm94LXNpemluZzpib3JkZXItYm94O2ZvbnQtc2l6ZTowO3RleHQtYWxpZ246Y2VudGVyIj48ZGl2IHN0eWxlPSJkaXNwbGF5OmlubGluZS1ibG9jaztmb250LXNpemU6MTFweDtmb250LWZhbWlseTpWZXJkYW5hO2NvbG9yOiMwMDA7bGluZS1oZWlnaHQ6MS4yO3BvaW50ZXItZXZlbnRzOm5vbmU7d2hpdGUtc3BhY2U6bm9ybWFsO292ZXJmbG93LXdyYXA6bm9ybWFsIj5EZXN0aW5hdGlvbjogRWdyZXNzIElQPC9kaXY+PC9kaXY+PC9kaXY+PC9mb3JlaWduT2JqZWN0Pjx0ZXh0IHg9IjQ0MCIgeT0iMTE1IiBmb250LWZhbWlseT0iVmVyZGFuYSIgZm9udC1zaXplPSIxMSIgdGV4dC1hbmNob3I9Im1pZGRsZSI+RGVzdGluYXRpb24uLi48L3RleHQ+PC9zd2l0Y2g+PHBhdGggZmlsbD0ibm9uZSIgc3Ryb2tlPSIjMDAwIiBzdHJva2UtbWl0ZXJsaW1pdD0iMTAiIGQ9Ik00MDAgOTRoOTBNMjM2LjM3IDk1LjggMjU5IDk2aDcwIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxwYXRoIHN0cm9rZT0iIzAwMCIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBkPSJtMjMxLjEyIDk1Ljc2IDcuMDMtMy40NC0xLjc4IDMuNDggMS43MiAzLjUyWiIgcG9pbnRlci1ldmVudHM9Im5vbmUiLz48c3dpdGNoPjxnIHJlcXVpcmVkRmVhdHVyZXM9Imh0dHA6Ly93d3cudzMub3JnL1RSL1NWRzExL2ZlYXR1cmUjRXh0ZW5zaWJpbGl0eSIvPjxhIHhsaW5rOmhyZWY9Imh0dHBzOi8vd3d3LmRpYWdyYW1zLm5ldC9kb2MvZmFxL3N2Zy1leHBvcnQtdGV4dC1wcm9ibGVtcyIgdGFyZ2V0PSJfYmxhbmsiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDAgLTUpIj48dGV4dCB4PSI1MCUiIHk9IjEwMCUiIGZvbnQtc2l6ZT0iMTAiIHRleHQtYW5jaG9yPSJtaWRkbGUiPlZpZXdlciBkb2VzIG5vdCBzdXBwb3J0IGZ1bGwgU1ZHIDEuMTwvdGV4dD48L2E+PC9zd2l0Y2g+PC9zdmc+) Options to make routers aware of egress IP: - Program routes statically on routers - Peer routers with the cluster, directly or indirectly using BGP or some other protocol, or other method so routers learn about the egress IP ### Does your egress gateway have required metadata?[​](#does-your-egress-gateway-have-required-metadata) Review important egress gateway metadata (for example, namespace and labels); they are required for a client to identify the gateway(s) that it should use. ### Is natOutgoing on your IPPool set up correctly?[​](#is-natoutgoing-on-your-ippool-set-up-correctly) For most egress gateway scenarios you should have: `natOutgoing: false` on the egress IPPool. If you have `natOutgoing: true`, the egress gateway will SNAT to its own IP, which is the intended egress gateway IP. But then the egress gateway’s node will also SNAT to its own IP (i.e. the node IP), which immediately overrides the egress gateway IP. ### Do clients and nodes have required selectors?[​](#do-clients-and-nodes-have-required-selectors) Review the following annotations that are required for the client to identify its egress gateways: - egress.projectcalico.org/egressGatewayPolicy - egress.projectcalico.org/selector - egress.projectcalico.org/namespaceSelector on - Client pod - Client pod’s namespace ### Does Calico have enough route tables?[​](#does-calico-have-enough-route-tables) On Linux, Calico claims a number of routing tables for various purposes. The exact number of tables allocated is determined by the [FelixConfiguration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig) option `routeTableRanges`. Failing to allocate enough route tables will result in Calico failing to program the networking needed for Egress Gateways to function. Under these conditions, `calico-node` pods will log error messages and report [unhealthy](#check-calico-node-health). [See more about routeTableRanges](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig#routetableranges) ### Check egress gateway health[​](#check-egress-gateway-health) As of v3.15.0, the egress gateway `Deployment` includes a Kubernetes `readinessProbe`. The egress gateway will only report ready if it is able to connect to the local `calico-node` pod and if any configured HTTP/ICMP probes are succeeding. If the egress gateway reports not-Ready then more information can be found in its logs. The egress gateway logs to `stdout` so its logs are available via `kubectl logs -n `. ### Check health of calico-node to egress gateway probes[​](#check-health-of-calico-node-to-egress-gateway-probes) As of v3.15.0, each `calico-node` pod probes the health of the remote egress gateways that its local pods are using. If probes fail, the failures are logged in `calico-node`'s log (search for `egress_gw_tracker`) and reported via Prometheus metrics: ```text felix_egress_gateway_remote_polls{status="probe-failed"} 0 felix_egress_gateway_remote_polls{status="total"} 2 felix_egress_gateway_remote_polls{status="up"} 2 ``` Where, the `total` metric reports the total number of remote egress gateways that are being polled and the `up` and `probe-failed` metrics report the number of egress gateways in each of those states. ### Check calico-node health[​](#check-calico-node-health) Check that your calico-node pods are consistently running and ready, especially on the nodes hosting the client and gateway pods. Confirming healthy pods will rule out possible issues. If you find that `calico-node` is not ready, describing the pod should show which health check is failing: ```bash kubectl describe pod ... ``` In AWS, issues such as permissions problems will report a problem with the `"aws-eni-addresses-in-sync"` health check. For more information on the problem, review the `calico-node` log. For example, a permissions issue will result in a log such as the following: ```text 2021-11-16 13:11:59.292 [ERROR][26606] felix/aws_iface_provisioner.go 343: Failed to resync with AWS. Will retry after backoff. error=failed to create ENI: operation error EC2: CreateNetworkInterface, https response error StatusCode: 403, RequestID: 13dead98-7da0-4695-9be8-80cab4d5528e, api error UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: j4x3cFwZdJ......ShGkw ``` If you see such a log, check the AWS IAM permissions assigned to the nodes in your cluster to ensure that the nodes have the [required permissions](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-aws#configure-aws-iam-roles). It is also possible to decode the "encoded authorization failure message" in the log by following [this guide](https://aws.amazon.com/premiumsupport/knowledge-center/aws-backup-encoded-authorization-failure/); this gives more detail on the error. ### Check IP rule and routing setup on the client node[​](#check-ip-rule-and-routing-setup-on-the-client-node) **Run `ip rule`** On the client node, run: ```text ip rule ``` **Sample output** You will see a line for each pod on the node that is configured to use an egress gateway. ```text from 192.168.24.35 fwmark 0x80000/0x80000 lookup 250 ``` Where: - `192.168.24.35` is the relevant client's pod IP - `250` is the routing table number - `fwmark 0x80000/0x80000` is the bit/mask If you don’t see this, it means one of the following: - egressIPSupport is not enabled - egressIPSupport is enabled, but you have not configured egress annotations on the client pod or on its namespace - egressIPSupport is EnabledPerNamespace and you have configured egress annotations on the client pod, but not on its namespace **Run `ip route show table`** On the client node, run the following command using the routing table number from the `ip rule` command. For example: `250`. ```text ip route show table ``` **Sample output: clients using a single egress gateway** ```text default via 11.11.11.1 dev egress.calico onlink ``` **Sample: clients using multiple gateways** ```text default onlink nexthop via 11.11.11.1 dev egress.calico weight 1 onlink nexthop via 11.11.11.2 dev egress.calico weight 1 onlink ``` **Sample: clients using the following egress gateway policy** ```yaml apiVersion: projectcalico.org/v3 kind: EgressGatewayPolicy metadata: name: "egw-policy1" spec: rules: - destination: cidr: 10.0.0.0/8 description: "Local: no gateway" - destination: cidr: 11.0.0.0/8 description: "Gateway to on prem" gateway: namespaceSelector: "projectcalico.org/name == 'default'" selector: "egress-code == 'blue'" maxNextHops: 2 - description: "Gateway to internet" gateway: namespaceSelector: "projectcalico.org/name == 'default'" selector: "egress-code == 'red'" ``` ```text default onlink nexthop via 11.11.11.1 dev egress.calico weight 1 onlink nexthop via 11.11.11.2 dev egress.calico weight 1 onlink nexthop via 11.11.11.3 dev egress.calico weight 1 onlink throw 10.0.0.0/8 11.0.0.0/8 onlink nexthop via 11.11.12.1 dev egress.calico weight 1 onlink nexthop via 11.11.12.2 dev egress.calico weight 1 onlink ``` If you see nothing at all, or the following: ```text unreachable default scope link ``` - Verify that you have provisioned the gateways - Review the selectors, and gateway namespace and labels to determine why they aren’t matching each other ### Do you have egress IPs in BGPConfiguration svcExternalIPs?[​](#do-you-have-egress-ips-in-bgpconfiguration-svcexternalips) You should not have any egress IPs or pod IP ranges in BGPConfiguration `serviceExternalIPs` or `serviceClusterIPs` fields; it causes problems if you do. By default, Calico Cloud BGP exports all pod IPs, which includes egress gateway IPs because they are pod IPs. But you can also use [BGPConfiguration resource parameters](https://docs.tigera.io/calico-cloud/reference/resources/bgpconfig) like `BGPConfiguration`, `serviceClusterIPs`, `serviceExternalIPs` and `serviceLoadBalancerIPs` to export additional IP ranges, in particular Kubernetes Service IPs. Because Calico Cloud exports additional IP ranges in a different way from pod IPs, things can go wrong if you include pod IPs in the additional ranges. ## Connection to an egress gateway cannot be established[​](#connection-to-an-egress-gateway-cannot-be-established) If the outbound connection cannot be established, the policy may be denying the flow. As shown in the following diagram, policy is enforced at more points in an egress gateway flow. ![egress-basic-routing](data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnhsaW5rPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rIiB3aWR0aD0iNjExIiBoZWlnaHQ9IjEyOCIgc3R5bGU9ImJhY2tncm91bmQtY29sb3I6I2ZmZiIgdmlld0JveD0iLTAuNSAtMC41IDYxMSAxMjgiPjxyZWN0IHdpZHRoPSIyNDAiIGhlaWdodD0iMTE4IiB5PSI5IiBmaWxsPSIjOGViYmZjIiBmaWxsLW9wYWNpdHk9Ii4zIiBzdHJva2U9IiNkZGQiIHN0cm9rZS1vcGFjaXR5PSIuMyIgcG9pbnRlci1ldmVudHM9Im5vbmUiIHJ4PSIxIiByeT0iMSIvPjxwYXRoIGZpbGw9IiMwNmMiIGQ9Ik04NC4wNCA0NkM3NS45IDQ2IDY5IDM5LjY2IDY5IDMwLjkgNjkgMjIuNDIgNzYuMDcgMTYgODQuMDEgMTYgOTIuMjUgMTYgOTkgMjMuMTkgOTkgMzAuOTMgOTkgMzkuMDkgOTIuMzYgNDYgODQuMDQgNDYiIHBvaW50ZXItZXZlbnRzPSJub25lIi8+PHBhdGggZmlsbD0iI2ZmZiIgZD0iTTc4LjYgMjYuOTFjLS44IDAtMS4zMi0uNjItMS4zMi0xLjM4IDAtLjY0LjUyLTEuMyAxLjMyLTEuMy43NiAwIDEuMzYuNjEgMS4zNiAxLjMgMCAuNzItLjUyIDEuMzgtMS4zNiAxLjM4bTEwLjc0IDBjLS44IDAtMS4zMi0uNjItMS4zMi0xLjM4IDAtLjY0LjUyLTEuMyAxLjMyLTEuMy43NiAwIDEuMzYuNjEgMS4zNiAxLjMgMCAuNzItLjUyIDEuMzgtMS4zNiAxLjM4bTAgMTAuNzhjLS44IDAtMS4zMi0uNjItMS4zMi0xLjM4IDAtLjY0LjUyLTEuMyAxLjMyLTEuMy43NiAwIDEuMzYuNjEgMS4zNiAxLjMgMCAuNzItLjUyIDEuMzgtMS4zNiAxLjM4bS0xMi4wNi0zLjkyYy0xLjU0IDAtMi44LTEuMjEtMi44LTIuNzkgMC0xLjIxIDEtMi43OCAyLjgtMi43OCAxLjU4IDAgMi43OCAxLjI1IDIuNzggMi43OGEyLjc3IDIuNzcgMCAwIDEtMi43OCAyLjc5bTYuNzQtNi43NWMtMS41NSAwLTIuODEtMS4yMS0yLjgxLTIuNzkgMC0xLjIyIDEuMDEtMi43OCAyLjgxLTIuNzggMS41NyAwIDIuNzggMS4yNSAyLjc4IDIuNzhhMi43NyAyLjc3IDAgMCAxLTIuNzggMi43OW02LjcxIDYuNzVjLTEuNTUgMC0yLjgxLTEuMjEtMi44MS0yLjc5IDAtMS4yMSAxLjAxLTIuNzggMi44MS0yLjc4IDEuNTcgMCAyLjc4IDEuMjUgMi43OCAyLjc4YTIuNzcgMi43NyAwIDAgMS0yLjc4IDIuNzltLTYuNzQtLjY5Yy0xLjEzIDAtMi4wOS0uODctMi4wOS0yLjEgMC0xLjA2LjgzLTIuMDkgMi4xLTIuMDkgMS4wOSAwIDIuMTEuODQgMi4xMSAyLjEgMCAxLjIxLS45NSAyLjA5LTIuMTIgMi4wOW0tNS4zOSA0LjYxYy0uOCAwLTEuMzItLjYyLTEuMzItMS4zOCAwLS42NC41Mi0xLjMgMS4zMi0xLjMuNzYgMCAxLjM2LjYxIDEuMzYgMS4zIDAgLjcyLS41MiAxLjM4LTEuMzYgMS4zOG01LjQyIDIuODZjLTEuNTUgMC0yLjgxLTEuMjEtMi44MS0yLjc5IDAtMS4yMSAxLjAxLTIuNzggMi44MS0yLjc4IDEuNTcgMCAyLjc4IDEuMjUgMi43OCAyLjc4YTIuNzcgMi43NyAwIDAgMS0yLjc4IDIuNzkiIHBvaW50ZXItZXZlbnRzPSJub25lIi8+PHJlY3Qgd2lkdGg9IjkwIiBoZWlnaHQ9IjQ1IiB4PSIxMCIgeT0iNjIiIGZpbGw9IiNmZmYiIHBvaW50ZXItZXZlbnRzPSJub25lIiByeD0iNi43NSIgcnk9IjYuNzUiLz48c3dpdGNoIHRyYW5zZm9ybT0idHJhbnNsYXRlKC0uNSAtLjUpIj48Zm9yZWlnbk9iamVjdCB3aWR0aD0iMTAwJSIgaGVpZ2h0PSIxMDAlIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIgcmVxdWlyZWRGZWF0dXJlcz0iaHR0cDovL3d3dy53My5vcmcvVFIvU1ZHMTEvZmVhdHVyZSNFeHRlbnNpYmlsaXR5IiBzdHlsZT0ib3ZlcmZsb3c6dmlzaWJsZTt0ZXh0LWFsaWduOmxlZnQiPjxkaXYgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkveGh0bWwiIHN0eWxlPSJkaXNwbGF5OmZsZXg7YWxpZ24taXRlbXM6dW5zYWZlIGNlbnRlcjtqdXN0aWZ5LWNvbnRlbnQ6dW5zYWZlIGNlbnRlcjt3aWR0aDo4OHB4O2hlaWdodDoxcHg7cGFkZGluZy10b3A6ODVweDttYXJnaW4tbGVmdDoxMXB4Ij48ZGl2IHN0eWxlPSJib3gtc2l6aW5nOmJvcmRlci1ib3g7Zm9udC1zaXplOjA7dGV4dC1hbGlnbjpjZW50ZXIiPjxkaXYgc3R5bGU9ImRpc3BsYXk6aW5saW5lLWJsb2NrO2ZvbnQtc2l6ZToxMnB4O2ZvbnQtZmFtaWx5OlZlcmRhbmE7Y29sb3I6IzAwMDtsaW5lLWhlaWdodDoxLjI7cG9pbnRlci1ldmVudHM6bm9uZTtmb250LXdlaWdodDo3MDA7d2hpdGUtc3BhY2U6bm9ybWFsO292ZXJmbG93LXdyYXA6bm9ybWFsIj48Zm9udCBzdHlsZT0iZm9udC1zaXplOjExcHg7Zm9udC13ZWlnaHQ6NDAwIj5DbGllbnQ8L2ZvbnQ+PC9kaXY+PC9kaXY+PC9kaXY+PC9mb3JlaWduT2JqZWN0Pjx0ZXh0IHg9IjU1IiB5PSI4OCIgZm9udC1mYW1pbHk9IlZlcmRhbmEiIGZvbnQtc2l6ZT0iMTIiIGZvbnQtd2VpZ2h0PSJib2xkIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5DbGllbnQ8L3RleHQ+PC9zd2l0Y2g+PHJlY3Qgd2lkdGg9IjkwIiBoZWlnaHQ9IjQ1IiB4PSIxNDAiIHk9IjYyIiBmaWxsPSIjZmZmIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIgcng9IjYuNzUiIHJ5PSI2Ljc1Ii8+PHN3aXRjaCB0cmFuc2Zvcm09InRyYW5zbGF0ZSgtLjUgLS41KSI+PGZvcmVpZ25PYmplY3Qgd2lkdGg9IjEwMCUiIGhlaWdodD0iMTAwJSIgcG9pbnRlci1ldmVudHM9Im5vbmUiIHJlcXVpcmVkRmVhdHVyZXM9Imh0dHA6Ly93d3cudzMub3JnL1RSL1NWRzExL2ZlYXR1cmUjRXh0ZW5zaWJpbGl0eSIgc3R5bGU9Im92ZXJmbG93OnZpc2libGU7dGV4dC1hbGlnbjpsZWZ0Ij48ZGl2IHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hodG1sIiBzdHlsZT0iZGlzcGxheTpmbGV4O2FsaWduLWl0ZW1zOnVuc2FmZSBjZW50ZXI7anVzdGlmeS1jb250ZW50OnVuc2FmZSBjZW50ZXI7d2lkdGg6ODhweDtoZWlnaHQ6MXB4O3BhZGRpbmctdG9wOjg1cHg7bWFyZ2luLWxlZnQ6MTQxcHgiPjxkaXYgc3R5bGU9ImJveC1zaXppbmc6Ym9yZGVyLWJveDtmb250LXNpemU6MDt0ZXh0LWFsaWduOmNlbnRlciI+PGRpdiBzdHlsZT0iZGlzcGxheTppbmxpbmUtYmxvY2s7Zm9udC1zaXplOjEycHg7Zm9udC1mYW1pbHk6VmVyZGFuYTtjb2xvcjojMDAwO2xpbmUtaGVpZ2h0OjEuMjtwb2ludGVyLWV2ZW50czpub25lO2ZvbnQtd2VpZ2h0OjcwMDt3aGl0ZS1zcGFjZTpub3JtYWw7b3ZlcmZsb3ctd3JhcDpub3JtYWwiPjxmb250IHN0eWxlPSJmb250LXNpemU6MTFweDtmb250LXdlaWdodDo0MDAiPkdhdGV3YXk8L2ZvbnQ+PC9kaXY+PC9kaXY+PC9kaXY+PC9mb3JlaWduT2JqZWN0Pjx0ZXh0IHg9IjE4NSIgeT0iODgiIGZvbnQtZmFtaWx5PSJWZXJkYW5hIiBmb250LXNpemU9IjEyIiBmb250LXdlaWdodD0iYm9sZCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+R2F0ZXdheTwvdGV4dD48L3N3aXRjaD48cmVjdCB3aWR0aD0iMTIwIiBoZWlnaHQ9IjgwIiB4PSI0OTAiIHk9IjM3IiBmaWxsPSIjZmZmIiBzdHJva2U9IiM3ZWE2ZTAiIHBvaW50ZXItZXZlbnRzPSJub25lIiByeD0iMTIiIHJ5PSIxMiIvPjxzd2l0Y2ggdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLS41IC0uNSkiPjxmb3JlaWduT2JqZWN0IHdpZHRoPSIxMDAlIiBoZWlnaHQ9IjEwMCUiIHBvaW50ZXItZXZlbnRzPSJub25lIiByZXF1aXJlZEZlYXR1cmVzPSJodHRwOi8vd3d3LnczLm9yZy9UUi9TVkcxMS9mZWF0dXJlI0V4dGVuc2liaWxpdHkiIHN0eWxlPSJvdmVyZmxvdzp2aXNpYmxlO3RleHQtYWxpZ246bGVmdCI+PGRpdiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94aHRtbCIgc3R5bGU9ImRpc3BsYXk6ZmxleDthbGlnbi1pdGVtczp1bnNhZmUgY2VudGVyO2p1c3RpZnktY29udGVudDp1bnNhZmUgY2VudGVyO3dpZHRoOjExOHB4O2hlaWdodDoxcHg7cGFkZGluZy10b3A6NzdweDttYXJnaW4tbGVmdDo0OTFweCI+PGRpdiBzdHlsZT0iYm94LXNpemluZzpib3JkZXItYm94O2ZvbnQtc2l6ZTowO3RleHQtYWxpZ246Y2VudGVyIj48ZGl2IHN0eWxlPSJkaXNwbGF5OmlubGluZS1ibG9jaztmb250LXNpemU6MTJweDtmb250LWZhbWlseTpWZXJkYW5hO2NvbG9yOiMwMDA7bGluZS1oZWlnaHQ6MS4yO3BvaW50ZXItZXZlbnRzOm5vbmU7Zm9udC13ZWlnaHQ6NzAwO3doaXRlLXNwYWNlOm5vcm1hbDtvdmVyZmxvdy13cmFwOm5vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtd2VpZ2h0OjQwMCI+RGVzdGluYXRpb248L3NwYW4+PC9kaXY+PC9kaXY+PC9kaXY+PC9mb3JlaWduT2JqZWN0Pjx0ZXh0IHg9IjU1MCIgeT0iODEiIGZvbnQtZmFtaWx5PSJWZXJkYW5hIiBmb250LXNpemU9IjEyIiBmb250LXdlaWdodD0iYm9sZCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+RGVzdGluYXRpb248L3RleHQ+PC9zd2l0Y2g+PHN3aXRjaCB0cmFuc2Zvcm09InRyYW5zbGF0ZSgtLjUgLS41KSI+PGZvcmVpZ25PYmplY3Qgd2lkdGg9IjEwMCUiIGhlaWdodD0iMTAwJSIgcG9pbnRlci1ldmVudHM9Im5vbmUiIHJlcXVpcmVkRmVhdHVyZXM9Imh0dHA6Ly93d3cudzMub3JnL1RSL1NWRzExL2ZlYXR1cmUjRXh0ZW5zaWJpbGl0eSIgc3R5bGU9Im92ZXJmbG93OnZpc2libGU7dGV4dC1hbGlnbjpsZWZ0Ij48ZGl2IHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hodG1sIiBzdHlsZT0iZGlzcGxheTpmbGV4O2FsaWduLWl0ZW1zOnVuc2FmZSBjZW50ZXI7anVzdGlmeS1jb250ZW50OnVuc2FmZSBjZW50ZXI7d2lkdGg6NThweDtoZWlnaHQ6MXB4O3BhZGRpbmctdG9wOjMycHg7bWFyZ2luLWxlZnQ6MTAwcHgiPjxkaXYgc3R5bGU9ImJveC1zaXppbmc6Ym9yZGVyLWJveDtmb250LXNpemU6MDt0ZXh0LWFsaWduOmNlbnRlciI+PGRpdiBzdHlsZT0iZGlzcGxheTppbmxpbmUtYmxvY2s7Zm9udC1zaXplOjEycHg7Zm9udC1mYW1pbHk6VmVyZGFuYTtjb2xvcjojMDAwO2xpbmUtaGVpZ2h0OjEuMjtwb2ludGVyLWV2ZW50czpub25lO2ZvbnQtd2VpZ2h0OjcwMDt3aGl0ZS1zcGFjZTpub3JtYWw7b3ZlcmZsb3ctd3JhcDpub3JtYWwiPkNsdXN0ZXI8L2Rpdj48L2Rpdj48L2Rpdj48L2ZvcmVpZ25PYmplY3Q+PHRleHQgeD0iMTI5IiB5PSIzNiIgZm9udC1mYW1pbHk9IlZlcmRhbmEiIGZvbnQtc2l6ZT0iMTIiIGZvbnQtd2VpZ2h0PSJib2xkIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5DbHVzdGVyPC90ZXh0Pjwvc3dpdGNoPjxwYXRoIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzAwMCIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBkPSJNMTA2LjM3IDg0aDI3LjI2IiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxwYXRoIHN0cm9rZT0iIzAwMCIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBkPSJtMTAxLjEyIDg0IDctMy41LTEuNzUgMy41IDEuNzUgMy41Wk0xMzguODggODRsLTcgMy41IDEuNzUtMy41LTEuNzUtMy41WiIgcG9pbnRlci1ldmVudHM9Im5vbmUiLz48cGF0aCBmaWxsPSIjN2VhNmUwIiBkPSJNMzY0Ljc2IDExN0MzNDMuNTUgMTE3IDMyOSAxMDEgMzI5IDg0LjQydi0zLjcxYzAtMTYuMTcgMTQuMDctMzIuNjggMzUuNTEtMzIuNjggMTguOTYgMCAzNS40NyAxNC4wMyAzNS40NyAzMy4wNGwuMDIgMi44OWMwIDE4Ljc4LTE2LjQ0IDMzLjA0LTM1LjI0IDMzLjA0IiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxwYXRoIGZpbGwtb3BhY2l0eT0iLjMiIGQ9Ik0zNjQuNjYgMTE0LjE3YzE5LjAxIDAgMzUuMzItMTQuMzcgMzUuMzItMzMuMWwuMDIgMi44OWMwIDE4LjgtMTYuNDUgMzMuMDQtMzUuMjQgMzMuMDQtMjEuMjggMC0zNS43Ni0xNS45OC0zNS43Ni0zMi41OHYtMy43MWMwIDE5Ljk3IDE3LjMxIDMzLjQ2IDM1LjY2IDMzLjQ2IiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxwYXRoIGZpbGw9IiNmZmYiIGQ9Im0zNjQuNDkgMTEwLjkxLTcuMjQtMTEuNzVoNC4zMlY4NC4wM2gtMTYuMjJ2NC4wMmwtMTIuNjEtNi43MyAxMi42MS02Ljg2djQuMTNoMTYuMjJWNjMuMzZoLTQuMzJsNy4yNC0xMS43NCA3LjI0IDExLjc0aC00LjMxdjE1LjIzaDE2LjIydi00LjEzbDEyLjYyIDYuODYtMTIuNjIgNi43NHYtNC4wM2gtMTYuMjJ2MTUuMTNoNC4zMVoiIHBvaW50ZXItZXZlbnRzPSJub25lIi8+PHN3aXRjaCB0cmFuc2Zvcm09InRyYW5zbGF0ZSgtLjUgLS41KSI+PGZvcmVpZ25PYmplY3Qgd2lkdGg9IjEwMCUiIGhlaWdodD0iMTAwJSIgcG9pbnRlci1ldmVudHM9Im5vbmUiIHJlcXVpcmVkRmVhdHVyZXM9Imh0dHA6Ly93d3cudzMub3JnL1RSL1NWRzExL2ZlYXR1cmUjRXh0ZW5zaWJpbGl0eSIgc3R5bGU9Im92ZXJmbG93OnZpc2libGU7dGV4dC1hbGlnbjpsZWZ0Ij48ZGl2IHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hodG1sIiBzdHlsZT0iZGlzcGxheTpmbGV4O2FsaWduLWl0ZW1zOnVuc2FmZSBjZW50ZXI7anVzdGlmeS1jb250ZW50OnVuc2FmZSBjZW50ZXI7d2lkdGg6NjlweDtoZWlnaHQ6MXB4O3BhZGRpbmctdG9wOjM0cHg7bWFyZ2luLWxlZnQ6MzMwcHgiPjxkaXYgc3R5bGU9ImJveC1zaXppbmc6Ym9yZGVyLWJveDtmb250LXNpemU6MDt0ZXh0LWFsaWduOmNlbnRlciI+PGRpdiBzdHlsZT0iZGlzcGxheTppbmxpbmUtYmxvY2s7Zm9udC1zaXplOjEycHg7Zm9udC1mYW1pbHk6VmVyZGFuYTtjb2xvcjojMDAwO2xpbmUtaGVpZ2h0OjEuMjtwb2ludGVyLWV2ZW50czpub25lO2ZvbnQtd2VpZ2h0OjcwMDt3aGl0ZS1zcGFjZTpub3JtYWw7b3ZlcmZsb3ctd3JhcDpub3JtYWwiPlJvdXRlcnM8L2Rpdj48L2Rpdj48L2Rpdj48L2ZvcmVpZ25PYmplY3Q+PHRleHQgeD0iMzY1IiB5PSIzOCIgZm9udC1mYW1pbHk9IlZlcmRhbmEiIGZvbnQtc2l6ZT0iMTIiIGZvbnQtd2VpZ2h0PSJib2xkIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5Sb3V0ZXJzPC90ZXh0Pjwvc3dpdGNoPjxwYXRoIGZpbGw9Im5vbmUiIHN0cm9rZT0iIzAwMCIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBkPSJNMjMwIDc2LjVoMTAwIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxzd2l0Y2ggdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLS41IC0uNSkiPjxmb3JlaWduT2JqZWN0IHdpZHRoPSIxMDAlIiBoZWlnaHQ9IjEwMCUiIHBvaW50ZXItZXZlbnRzPSJub25lIiByZXF1aXJlZEZlYXR1cmVzPSJodHRwOi8vd3d3LnczLm9yZy9UUi9TVkcxMS9mZWF0dXJlI0V4dGVuc2liaWxpdHkiIHN0eWxlPSJvdmVyZmxvdzp2aXNpYmxlO3RleHQtYWxpZ246bGVmdCI+PGRpdiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94aHRtbCIgc3R5bGU9ImRpc3BsYXk6ZmxleDthbGlnbi1pdGVtczp1bnNhZmUgY2VudGVyO2p1c3RpZnktY29udGVudDp1bnNhZmUgY2VudGVyO3dpZHRoOjU4cHg7aGVpZ2h0OjFweDtwYWRkaW5nLXRvcDo2MnB4O21hcmdpbi1sZWZ0OjI2MXB4Ij48ZGl2IHN0eWxlPSJib3gtc2l6aW5nOmJvcmRlci1ib3g7Zm9udC1zaXplOjA7dGV4dC1hbGlnbjpjZW50ZXIiPjxkaXYgc3R5bGU9ImRpc3BsYXk6aW5saW5lLWJsb2NrO2ZvbnQtc2l6ZToxMXB4O2ZvbnQtZmFtaWx5OlZlcmRhbmE7Y29sb3I6IzAwMDtsaW5lLWhlaWdodDoxLjI7cG9pbnRlci1ldmVudHM6bm9uZTt3aGl0ZS1zcGFjZTpub3JtYWw7b3ZlcmZsb3ctd3JhcDpub3JtYWwiPlNvdXJjZTogRWdyZXNzIElQPC9kaXY+PC9kaXY+PC9kaXY+PC9mb3JlaWduT2JqZWN0Pjx0ZXh0IHg9IjI5MCIgeT0iNjUiIGZvbnQtZmFtaWx5PSJWZXJkYW5hIiBmb250LXNpemU9IjExIiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5Tb3VyY2U6IEVnci4uLjwvdGV4dD48L3N3aXRjaD48cGF0aCBmaWxsPSJub25lIiBzdHJva2U9IiMwMDAiIHN0cm9rZS1taXRlcmxpbWl0PSIxMCIgZD0iTTQwMCA3Ni41aDgzLjYzIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxwYXRoIHN0cm9rZT0iIzAwMCIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBkPSJtNDg4Ljg4IDc2LjUtNyAzLjUgMS43NS0zLjUtMS43NS0zLjVaIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxzd2l0Y2ggdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLS41IC0uNSkiPjxmb3JlaWduT2JqZWN0IHdpZHRoPSIxMDAlIiBoZWlnaHQ9IjEwMCUiIHBvaW50ZXItZXZlbnRzPSJub25lIiByZXF1aXJlZEZlYXR1cmVzPSJodHRwOi8vd3d3LnczLm9yZy9UUi9TVkcxMS9mZWF0dXJlI0V4dGVuc2liaWxpdHkiIHN0eWxlPSJvdmVyZmxvdzp2aXNpYmxlO3RleHQtYWxpZ246bGVmdCI+PGRpdiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94aHRtbCIgc3R5bGU9ImRpc3BsYXk6ZmxleDthbGlnbi1pdGVtczp1bnNhZmUgY2VudGVyO2p1c3RpZnktY29udGVudDp1bnNhZmUgY2VudGVyO3dpZHRoOjU4cHg7aGVpZ2h0OjFweDtwYWRkaW5nLXRvcDoxMTJweDttYXJnaW4tbGVmdDo0MTFweCI+PGRpdiBzdHlsZT0iYm94LXNpemluZzpib3JkZXItYm94O2ZvbnQtc2l6ZTowO3RleHQtYWxpZ246Y2VudGVyIj48ZGl2IHN0eWxlPSJkaXNwbGF5OmlubGluZS1ibG9jaztmb250LXNpemU6MTFweDtmb250LWZhbWlseTpWZXJkYW5hO2NvbG9yOiMwMDA7bGluZS1oZWlnaHQ6MS4yO3BvaW50ZXItZXZlbnRzOm5vbmU7d2hpdGUtc3BhY2U6bm9ybWFsO292ZXJmbG93LXdyYXA6bm9ybWFsIj5EZXN0aW5hdGlvbjogRWdyZXNzIElQPC9kaXY+PC9kaXY+PC9kaXY+PC9mb3JlaWduT2JqZWN0Pjx0ZXh0IHg9IjQ0MCIgeT0iMTE1IiBmb250LWZhbWlseT0iVmVyZGFuYSIgZm9udC1zaXplPSIxMSIgdGV4dC1hbmNob3I9Im1pZGRsZSI+RGVzdGluYXRpb24uLi48L3RleHQ+PC9zd2l0Y2g+PHBhdGggZmlsbD0ibm9uZSIgc3Ryb2tlPSIjMDAwIiBzdHJva2UtbWl0ZXJsaW1pdD0iMTAiIGQ9Ik00MDAgOTRoOTBNMjM2LjM3IDk1LjggMjU5IDk2aDcwIiBwb2ludGVyLWV2ZW50cz0ibm9uZSIvPjxwYXRoIHN0cm9rZT0iIzAwMCIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBkPSJtMjMxLjEyIDk1Ljc2IDcuMDMtMy40NC0xLjc4IDMuNDggMS43MiAzLjUyWiIgcG9pbnRlci1ldmVudHM9Im5vbmUiLz48c3dpdGNoPjxnIHJlcXVpcmVkRmVhdHVyZXM9Imh0dHA6Ly93d3cudzMub3JnL1RSL1NWRzExL2ZlYXR1cmUjRXh0ZW5zaWJpbGl0eSIvPjxhIHhsaW5rOmhyZWY9Imh0dHBzOi8vd3d3LmRpYWdyYW1zLm5ldC9kb2MvZmFxL3N2Zy1leHBvcnQtdGV4dC1wcm9ibGVtcyIgdGFyZ2V0PSJfYmxhbmsiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDAgLTUpIj48dGV4dCB4PSI1MCUiIHk9IjEwMCUiIGZvbnQtc2l6ZT0iMTAiIHRleHQtYW5jaG9yPSJtaWRkbGUiPlZpZXdlciBkb2VzIG5vdCBzdXBwb3J0IGZ1bGwgU1ZHIDEuMTwvdGV4dD48L2E+PC9zd2l0Y2g+PC9zdmc+) **Policy enforcement**: - From the client pod, egress - To the gateway pod, ingress - From the gateway pod, egress - Any relevant HostEndpoints that are configured in your cluster In [the web console](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/tour), check for dropped packets because of policy on the outbound connection path. If you are using the iptables data plane, you can also run the following command on the client and gateway nodes to look at a lower level. ```text watch iptables-save -c | grep DROP | grep -v 0:0 ``` ## Connection to an egress gateway is established, but destination is not getting correct IP[​](#connection-to-an-egress-gateway-is-established-but-destination-is-not-getting-correct-ip) If you see that the outbound connection established, but the source IP is incorrect at the destination, this can indicate that other SNAT or MASQUERADE is taking effect after the packet leaves the egress gateway pod and is overriding the egress gateway IP. If you intentionally have a MASQUERADE/SNAT for another general purpose, you must filter it so it does not apply to traffic whose source IP comes from the egress gateway pool. To check the egress gateway’s node, use iptables: ```text iptables-save -c | grep -i MASQUERADE iptables-save -c | grep -i SNAT ``` ## Finding leaked AWS ENIs[​](#finding-leaked-aws-enis) In normal usage of the AWS-backed IP pools feature, the calico-node Pod on each node will manage the secondary ENIs used for networking AWS-backed IP pools. It also marks its secondary ENIs for deletion on instance termination to avoid leaking any ENIs when an instance is terminated. However, in certain highly unusual situations, such as the following sequence of events: - calico-node adds an ENI. - The AWS API call to mark the ENI for "delete on termination" fails. - The entire instance is deleted before the automatic retry of the above operation succeeds. Then, it would be possible for an ENI to be leaked. Calico Cloud marks all the ENIs that it creates with tags to identify them as Calico Cloud secondary ENIs and the ID of the instance they *should* belong to. To find potentially leaked ENIs, you can use the AWS command line tool as follows: ```bash aws ec2 describe-network-interfaces --filters Name=status,Values=available Name=tag-key,Values=calico:use ``` Then, examine the "Tag set" of the returned network interface values to see if any of them belong to nodes that have been deleted: ```text "TagSet": [ { "Key": "calico:use", "Value": "secondary" }, { "Key": "calico:instance", "Value": "i-00122bf604c6ab776" } ], ``` If the instance ID recorded in the "calico:instance" tag is for an instance that no longer exists then the ENI has been leaked; it is safe to delete the ENI. If the instance ID belongs to an active instance then there is no need to delete the ENI, it should be cleaned up (or put into use) by the calico-node Pod running on that instance. ### IP address management ## [📄️ Get started with IP address management](https://docs.tigera.io/calico-cloud/networking/ipam/get-started-ip-addresses) [Configure Calico Cloud to use Calico Cloud IPAM or host-local IPAM, and when to use one or the other.](https://docs.tigera.io/calico-cloud/networking/ipam/get-started-ip-addresses) ## [📄️ Configure default IP pools](https://docs.tigera.io/calico-cloud/networking/ipam/initial-ippool) [Configure the default IP address ranges for operator installation.](https://docs.tigera.io/calico-cloud/networking/ipam/initial-ippool) ## [📄️ Configure IP autodetection](https://docs.tigera.io/calico-cloud/networking/ipam/ip-autodetection) [Calico IP autodetection ensures the correct IP address is used for routing. Learn how to customize it.](https://docs.tigera.io/calico-cloud/networking/ipam/ip-autodetection) ## [📄️ Configure dual stack](https://docs.tigera.io/calico-cloud/networking/ipam/ipv6) [Configure dual stack for workloads.](https://docs.tigera.io/calico-cloud/networking/ipam/ipv6) ## [📄️ Use a specific IP address with a pod](https://docs.tigera.io/calico-cloud/networking/ipam/use-specific-ip) [Specify the IP address for a pod instead of allowing Calico Cloud to automatically choose one.](https://docs.tigera.io/calico-cloud/networking/ipam/use-specific-ip) ## [📄️ Assign IP addresses based on topology](https://docs.tigera.io/calico-cloud/networking/ipam/assign-ip-addresses-topology) [Configure Calico Cloud to use specific IP pools for different topologies including zone, rack, or region.](https://docs.tigera.io/calico-cloud/networking/ipam/assign-ip-addresses-topology) ## [📄️ Migrate from one IP pool to another](https://docs.tigera.io/calico-cloud/networking/ipam/migrate-pools) [Migrate pods from one IP pool to another on a running cluster without network disruption.](https://docs.tigera.io/calico-cloud/networking/ipam/migrate-pools) ## [📄️ Change IP pool block size](https://docs.tigera.io/calico-cloud/networking/ipam/change-block-size) [Expand or shrink the IP pool block size to efficiently manage IP pool addresses.](https://docs.tigera.io/calico-cloud/networking/ipam/change-block-size) ## [📄️ Restrict a pod to use an IP address in a specific range](https://docs.tigera.io/calico-cloud/networking/ipam/legacy-firewalls) [Restrict the IP address chosen for a pod to a specific range of IP addresses.](https://docs.tigera.io/calico-cloud/networking/ipam/legacy-firewalls) ## [📄️ LoadBalancer IP address management](https://docs.tigera.io/calico-cloud/networking/ipam/service-loadbalancer) [LoadBalancer IP address management](https://docs.tigera.io/calico-cloud/networking/ipam/service-loadbalancer) ### Get started with IP address management ## Big picture[​](#big-picture) Understand how IP address management (IPAM) functions in a Kubernetes cluster using Calico Cloud. ## Value[​](#value) Different IPAM techniques provide different feature sets. Calico Cloud’s IPAM provides additional IP allocation efficiency and flexibility compared to other address management approaches. ## Concepts[​](#concepts) ### IPAM in Kubernetes[​](#ipam-in-kubernetes) Kubernetes uses IPAM plugins to allocate and manage IP addresses assigned to pods. Different IPAM plugins provide different feature sets. Calico Cloud provides its own IPAM plugin called **calico-ipam** which is designed to work well with Calico Cloud and includes a number of features. ### Calico Cloud IPAM[​](#calico-cloud-ipam) The **calico-ipam** plugin uses Calico Cloud’s IP pool resource to control how IP addresses are allocated to pods within the cluster. This is the default plugin used by most Calico Cloud installations. By default, Calico Cloud uses a single IP pool for the entire Kubernetes pod CIDR, but you can divide the pod CIDR into several pools. You can assign separate IP pools to particular selections of **nodes**, or to teams, users, or applications within a cluster using **namespaces**. You can control which pools Calico Cloud uses for each pod using - node selectors - an annotation on the pod’s namespace, or - an annotation on the pod Calico Cloud also supports the **host-local** IPAM plugin. However, when using the host-local IPAM plugin some Calico Cloud features are not available. ### Calico Cloud IPAM blocks[​](#calico-cloud-ipam-blocks) In Calico Cloud IPAM, IP pools are subdivided into blocks -- smaller chunks that are associated with a particular node in the cluster. Each node in the cluster can have one or more blocks associated with it. Calico Cloud will automatically create and destroy blocks as needed as the number of nodes and pods in the cluster grows or shrinks. Blocks allow Calico Cloud to efficiently aggregate addresses assigned to pods on the same node, reducing the size of the routing table. By default Calico Cloud will try to allocate IP addresses from within an associated block, creating a new block if necessary. Calico Cloud can also assign addresses to pods on a node that are not within a block associated with that node. This allows for IP allocations independent of the node on which a pod is launched. By default, Calico Cloud creates blocks with room for 64 addresses (a /26), but you can control block sizes for each IP pool. ### Host-local IPAM[​](#host-local-ipam) The host-local plugin is a simple IP address management plugin. It uses predetermined CIDRs statically allocated to each node to choose addresses for pods. Once set, the CIDR for a node cannot be modified. Pods can be assigned addresses only from within the CIDR allocated to the node. Calico Cloud can use the host-local IPAM plugin, using the **Node.Spec.PodCIDR** field in the Kubernetes API to determine the CIDR to use for each node. However, per-node, per-pod, and per-namespace IP allocation features are not available using the host-local plugin. The host-local IPAM plugin is primarily used by other methods of routing pod traffic from one host to another. For example, it is used when installing Calico Cloud for policy enforcement with flannel networking, as well as when using Calico Cloud in Google Kubernetes Engine (GKE). ## Before you begin[​](#before-you-begin) **Limitations** - Works for platforms that use the Calico CNI ## How to[​](#how-to) ### Install Calico Cloud with calico-ipam[​](#install-calico-cloud-with-calico-ipam) Follow one of the getting started guides to install Calico Cloud. ### Install Calico Cloud with host-local IPAM[​](#install-calico-cloud-with-host-local-ipam) Follow one of the getting started guides to install Calico Cloud. Or, see the [reference documentation on host-local IPAM](https://docs.tigera.io/calico-cloud/reference/component-resources/configuration#using-host-local-ipam). ## Tutorial[​](#tutorial) For a blog/tutorial on IP pools, see [Calico Enterprise IPAM: Explained and Enhanced](https://www.tigera.io/blog/calico-ipam-explained-and-enhanced/). ## Additional resources[​](#additional-resources) - [IP Pool](https://docs.tigera.io/calico-cloud/reference/resources/ippool) There are several other ways to leverage Calico Cloud IPAM including: - [Assign addresses based on topology](https://docs.tigera.io/calico-cloud/networking/ipam/assign-ip-addresses-topology) - [Use a specific address for a pod](https://docs.tigera.io/calico-cloud/networking/ipam/use-specific-ip) - [Migrate from one IP pool to another](https://docs.tigera.io/calico-cloud/networking/ipam/migrate-pools) - [Restrict a pod to use an IP address in a specific range](https://docs.tigera.io/calico-cloud/networking/ipam/legacy-firewalls) - [Change IP address block size](https://docs.tigera.io/calico-cloud/reference/resources/ippool) ### Configure default IP pools ## Big picture[​](#big-picture) Configure the default IP pool values to use during Tigera Operator installation. ## Value[​](#value) During Tigera Operator installation, you must configure the CIDR range to use for pods that reflects your environment. ## Concepts[​](#concepts) ### Kubernetes pod CIDR[​](#kubernetes-pod-cidr) The **Kubernetes pod CIDR** is the expected IP address range for pod IPs. It is defined for the entire cluster, and is used by various Kubernetes components to determine if an IP belongs to a pod. For example, kube-proxy treats traffic differently if an IP is from a pod than if it is not. All pod IPs must be in the CIDR range for Kubernetes to function correctly. ### Tigera Operator and IP pools[​](#tigera-operator-and-ip-pools) [Calico IP pools](https://docs.tigera.io/calico-cloud/reference/resources/ippool) are ranges of IP addresses that Calico uses to assign to pods; the ranges must within the Kubernetes pod CIDR. The Tigera Operator reads the [Installation](https://docs.tigera.io/calico-cloud/reference/installation/api#installation) resource and configures the default Calico IP pool. Note the following: - Default fields for any that are omitted: - CIDR: 192.168.0.0/16 - Encapsulation: IPIP - NodeSelector: all() - NATOutgoing: Enabled - IP pools are only used when Calico is used for pod networking, IP pools are not utilized when using other pod networking solutions. - To make changes to the IP pools after Tigera Operator install, you may use calicoctl or kubectl. If you make the changes to the IP Pool in the Installation resource (Operator IPPool) after installation, the changes are not applied. ## Before you begin...[​](#before-you-begin) - Verify that your IP pool is within the Kubernetes pod CIDR - If you are using encapsulation (IP in IP or VXLAN), ensure that the traffic is allowed on your network - You are making these changes for a cluster that has not yet had Calico Cloud deployed. ## How to[​](#how-to) 1. Download the custom-resource.yaml file. 2. Edit the [Installation resource](https://docs.tigera.io/calico-cloud/reference/installation/api#installation). **Required values**: `cidr:`\ **Empty values**: Defaulted ```bash apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: calicoNetwork: ipPools: - cidr: "192.168.0.0/16" encapsulation: "IPIP" nodeSelector: "label == 'value'" natOutgoing: "Enabled" ``` 3. Apply the manifest and continue with your installation as normal. ## Additional resources[​](#additional-resources) - [IP pool resource](https://docs.tigera.io/calico-cloud/reference/resources/ippool) ### Configure IP autodetection ## Big picture[​](#big-picture) Configure IP autodetection for Calico Cloud nodes to ensure the correct IP address is used for routing. ## Value[​](#value) When you install Calico Cloud on a node, an IP address and subnet is automatically detected. Calico Cloud provides several ways to configure IP/subnet autodetection, and supports configuring specific IPs for: - Hosts with multiple external interfaces - Host interfaces with multiple IP addresses - [Changes to cross subnet packet encapsulation](https://docs.tigera.io/calico-cloud/networking/configuring/vxlan-ipip) - Changes to host IP address ## Concepts[​](#concepts) ### Autodetecting node IP address and subnet[​](#autodetecting-node-ip-address-and-subnet) For internode routing, each Calico Cloud node must be configured with an IPv4 address and/or an IPv6 address. When you install Calico Cloud on a node, a node resource is automatically created using routing information that is detected from the host. For some deployments, you may want to update autodetection to ensure nodes get the correct IP address. **Sample default node resource after installation** ```yaml apiVersion: projectcalico.org/v3 kind: Node metadata: name: node-hostname spec: bgp: asNumber: 64512 ipv4Address: 10.244.0.1/24 ipv6Address: 2000:db8:85a3::8a2e:370:7335/120 ipv4IPIPTunnelAddr: 192.168.0.1 ``` ### Autodetection methods[​](#autodetection-methods) By default, Calico Cloud uses the **firstFound** method; the first valid IP address on the first interface (excluding local interfaces such as the docker bridge). However, you can change the default method to any of the following: - Address used by the node to reach a particular IP or domain (**canReach**) - Address assigned to Kubernetes node (**kubernetes: InternalIP**) - Regex to include matching interfaces (**interface**) - Regex to exclude matching interfaces (**skipInterface**) - A list of IP ranges in CIDR format to determine valid IP addresses on the node to choose from (**cidrs**) For help on autodetection methods, see [NodeAddressAutodetection](https://docs.tigera.io/calico-cloud/reference/installation/api#nodeaddressautodetection) in the operator Installation reference and for more details see the [node configuration](https://docs.tigera.io/calico-cloud/reference/component-resources/node/configuration#ip-autodetection-methods) reference. ### Manually configure IP address and subnet[​](#manually-configure-ip-address-and-subnet) To manually configure an IP address and subnet, disable autodetection and update the node resources with the IP address. ## How to[​](#how-to) - [Change the autodetection method](#change-the-autodetection-method) - [Manually configure IP address and subnet for a node](#manually-configure-ip-address-and-subnet-for-a-node) ### Change the autodetection method[​](#change-the-autodetection-method) As noted previously, the default autodetection method is **first valid interface found** (firstFound). To use a different autodetection method, configure the NodeAddressAutodetection field(s) in the Installation resource. You can update the Installation resource before applying it during installation or edit it later with `kubectl edit installation default`. > **SECONDARY:** To configure the default autodetection method for IPv6 for any of the below methods, use the field `nodeAddressAutodetectionV6`. - **Kubernetes Node IP** Calico Cloud will select the first internal IP address listed in the Kubernetes node's `Status.Addresses` field. ```yaml kind: Installation apiVersion: operator.tigera.io/v1 metadata: name: default spec: calicoNetwork: nodeAddressAutodetectionV4: kubernetes: NodeInternalIP ``` - **Source address used to reach an IP or domain name** Calico Cloud will choose the IP address that is used to reach the given "can reach" IP address or domain. For example: ```yaml kind: Installation apiVersion: operator.tigera.io/v1 metadata: name: default spec: calicoNetwork: nodeAddressAutodetectionV4: canReach: 8.8.8.8 ``` - **Including matching interfaces** Calico Cloud will choose an address on each node from an interface that matches the given [regex](https://pkg.go.dev/regexp). For example: ```yaml kind: Installation apiVersion: operator.tigera.io/v1 metadata: name: default spec: calicoNetwork: nodeAddressAutodetectionV4: interface: eth.* ``` - **Excluding matching interfaces** Calico Cloud will choose an address on each node from an interface that does not match the given [regex](https://pkg.go.dev/regexp). For example: ```yaml kind: Installation apiVersion: operator.tigera.io/v1 metadata: name: default spec: calicoNetwork: nodeAddressAutodetectionV4: skipInterface: eth.* ``` - **Including CIDRs** Calico Cloud will select any IP address from the node that falls within the given CIDRs. For example: ```yaml kind: Installation apiVersion: operator.tigera.io/v1 metadata: name: default spec: calicoNetwork: nodeAddressAutodetectionV4: cidrs: - '192.168.200.0/24' ``` - **IPv4** ```yaml apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: variant: TigeraSecureEnterprise ... calicoNetwork: nodeAddressAutodetectionV4: : ``` - **IPv6** ```yaml apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: variant: TigeraSecureEnterprise ... calicoNetwork: nodeAddressAutodetectionV6: : ``` > **SECONDARY:** You can use both `nodeAddressAutodetectionV4` and `nodeAddressAutodetectionV6` to specify IPv4 and IPv6 methods. ### Manually configure IP address and subnet for a node[​](#manually-configure-ip-address-and-subnet-for-a-node) In the following scenarios, you may want to configure a specific IP and subnet: - Hosts with multiple external interfaces - Host interfaces with multiple IP addresses - Changes to cross subnet packet encapsulation - Changes to host IP address You can configure specific IP address and subnet for a node by disabling IP autodetection and then updating the [Node resource](https://docs.tigera.io/calico-cloud/reference/resources/node). #### Disable autodetection[​](#disable-autodetection) To disable autodetection method, update the proper `NodeAddressAutodetection` field in the Installation resource: ```yaml apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: variant: TigeraSecureEnterprise ... calicoNetwork: nodeAddressAutodetectionV4: {} # nodeAddressAutodetectionV6: {} ``` #### Configure IP and subnet using node resource[​](#configure-ip-and-subnet-using-node-resource) You can also configure the IP address and subnet on a Node resource. > **SECONDARY:** When configuring the IP address on a Node resource, you may want to disable IP address options or environment variables on the node. IP options on the container take precedence, and will overwrite the values you configure on the node resource. Use `calicoctl patch` to update the current node configuration. For example: ```text calicoctl patch node kind-control-plane \ --patch='{"spec":{"bgp": {"ipv4Address": "10.0.2.10/24", "ipv6Address": "fd80:24e2:f998:72d6::/120"}}}' ``` ## Additional resources[​](#additional-resources) - For details on autodetection methods, see the [node configuration](https://docs.tigera.io/calico-cloud/reference/component-resources/node/configuration) reference. - For calicoctl environment variables, see [Configuring cnx-node](https://docs.tigera.io/calico-cloud/reference/component-resources/node/configuration) - [Node resource](https://docs.tigera.io/calico-cloud/reference/resources/node) ### Configure dual stack ## Big picture[​](#big-picture) Configure Calico Cloud IP address allocation to use dual stack for workload communications. ## Value[​](#value) In addition to IPv4, IPv6 is increasingly desirable for Workload communication. Calico Cloud supports: - **IPv4 only** (default) Each workload gets an IPv4 address, and can communicate over IPv4. - **Dual stack** Each workload gets an IPv4 and an IPv6 address, and can communicate over IPv4 or IPv6. ## Before you begin[​](#before-you-begin) **Unsupported** - IPv6 only - AKS - AWS/kOps - EKS - GKE - RKE - RKE2 **Calico Cloud requirements** - Calico Cloud IPAM - OpenShift - Requires 4.8 for IPv6/IPv4 dual-stack and IPv6 single stack support - Requires 3.11 and later using Calico Cloud 3.4 and later for IPv6 support **Kubernetes version requirements** - For dual stack, 1.20 and later - For one IP stack at a time (IPv4 or IPv6), any Kubernetes version **Kubernetes IPv6 host requirements** - An IPv6 address that is reachable from the other hosts - The sysctl setting, `net.ipv6.conf.all.forwarding`, is set to `1`. This ensures both Kubernetes service traffic and Calico Cloud traffic is forwarded appropriately. - A default IPv6 route **Kubernetes IPv4 host requirements** - An IPv4 address that is reachable from the other hosts - The sysctl setting, `net.ipv4.conf.all.forwarding`, is set to `1`. This ensures both Kubernetes service traffic and Calico Cloud traffic is forwarded appropriately. - A default IPv4 route ## How to[​](#how-to) > **SECONDARY:** The following task is only for new clusters. ### Enable dual stack[​](#enable-dual-stack) 1. Set up a new cluster following the Kubernetes [prerequisites](https://kubernetes.io/docs/concepts/services-networking/dual-stack/#prerequisites) and [enablement steps](https://kubernetes.io/docs/concepts/services-networking/dual-stack/#enable-ipv4-ipv6-dual-stack). To configure dual-stack cluster using the operator, edit your default Installation at install time to include both an IPv4 and IPv6 pool. For example: ```yaml apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: # Configures Calico networking. calicoNetwork: ipPools: - blockSize: 26 cidr: 10.48.0.0/21 encapsulation: IPIP natOutgoing: Enabled nodeSelector: all() - blockSize: 122 cidr: 2001::00/64 encapsulation: None natOutgoing: Enabled nodeSelector: all() ``` ### Use a specific IP address with a pod ## Big picture[​](#big-picture) Choose the IP address for a pod instead of allowing Calico Cloud to choose automatically. ## Value[​](#value) Some applications require the use of stable IP addresses. Also, you may want to create entries in external DNS servers that point directly to pods, and this requires static IPs. ## Concepts[​](#concepts) ### Kubernetes pod CIDR[​](#kubernetes-pod-cidr) The **Kubernetes pod CIDR** is the range of IPs Kubernetes expects pod IPs to be assigned from. It is defined for the entire cluster and is used by various Kubernetes components to determine whether an IP belongs to a pod. For example, kube-proxy treats traffic differently if an IP is from a pod than if it is not. All pod IPs must be in the CIDR range for Kubernetes to function correctly. **IP Pools** IP pools are ranges of IP addresses from which Calico Cloud assigns pod IPs. Static IPs must be in an IP pool. ## Before you begin...[​](#before-you-begin) Your cluster must be using Calico IPAM to use this feature. If you are not sure which IPAM your cluster is using, the way to tell depends on install method. The IPAM plugin can be queried on the default Installation resource. ```text kubectl get installation default -o go-template --template {{.spec.cni.ipam.type}} ``` If your cluster is using Calico IPAM, the above command should return a result of `Calico`. ## How to[​](#how-to) Annotate the pod with cni.projectcalico.org/ipAddrs set to a list of IP addresses to assign, enclosed in brackets. For example: ```text "cni.projectcalico.org/ipAddrs": "[\"192.168.0.1\", \"2001:db8::1\"]" ``` Note the use of the escaped `\"` for the inner double quotes around the addresses. The address must be within a configured Calico Cloud IP pool and not currently in use. The annotation must be present when the pod is created; adding it later has no effect. Note that currently only a single IP address per IP family is supported per-pod using this annotation. ### Reserving IPs for manual assignments[​](#reserving-ips-for-manual-assignments) The `cni.projectcalico.org/ipAddrs` annotation requires the IP address to be within an IP pool. This means that, by default, Calico Cloud may decide to use the IP address that you select for another workload or for an internal tunnel address. To prevent this, there are several options: - To reserve a whole IPPool for manual allocations, you can set its node selector in the [IP pool](https://docs.tigera.io/calico-cloud/reference/resources/ippool) to `"!all()"`. Since the `!all()`\ cannot match any nodes, the IPPool will not be used for any automatic assignments. - To reserve part of a pool, you can create an [IP reservation](https://docs.tigera.io/calico-cloud/reference/resources/ipreservation). This allows for certain IPs to be reserved so that Calico IPAM will not use them automatically. However, manual assignments (using the annotation) can still use IPs that are "reserved". - To prevent Calico Cloud from using IPs from a certain pool for internal IPIP and/or VXLAN tunnel addresses, you can set the `allowedUses` field on the [IP Pool](https://docs.tigera.io/calico-cloud/reference/resources/ippool) to `["Workload"]`. ## Additional resources[​](#additional-resources) For help configuring Calico Cloud CNI and Calico Cloud IPAM, see [Configuring the Calico Cloud CNI Plugins](https://docs.tigera.io/calico-cloud/reference/component-resources/configuration). ### Assign IP addresses based on topology ## Big picture[​](#big-picture) Assign blocks of IP addresses from an IP pool for different topological areas. ## Value[​](#value) If you have workloads in different regions, zones, or rack, you may want them to get IP addresses from the same IP pool. This strategy is useful for reducing the number of routes that are required in the network, or to meet requirements imposed by an external firewall device or policy. Calico Cloud makes it easy to do this using an IP pool resource with node labels and node selectors. ## Concepts[​](#concepts) ### IP address assignment[​](#ip-address-assignment) Topology-based IP address assignment requires addresses to be per-host (node). As such, Kubernetes annotations cannot be used because annotations are only per-namespace and per-pod. And although you can configure IP addresses for nodes in the CNI configuration, you are making changes within the host’s file system. The best option is to use node-selection IP address assignment using IP pools. ### Node-selection IP address management[​](#node-selection-ip-address-management) Node selection-based IP address assignment is exactly what it sounds like: node labels are set, and Calico uses node selectors to decide which IP pools to use when assigning IP addresses to the node. ### Best practice[​](#best-practice) Nodes only assign workload addresses from IP pools which select them. To avoid having a workload not get an IP and fail to start, it is important to ensure that all nodes are selected by at least one IP pool. ## How to[​](#how-to) ### Create an IP pool, specific nodes[​](#create-an-ip-pool-specific-nodes) In the following example, we create an IP pool that only allocates IP addresses for nodes with the label, **zone=west**. ```yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: zone-west-ippool spec: cidr: 192.168.0.0/24 ipipMode: Always natOutgoing: true nodeSelector: zone == "west" ``` Then, we label a node with zone=west. For example: ```bash kubectl label nodes kube-node-0 zone=west ``` ## Tutorial[​](#tutorial) In this tutorial, we create a cluster with four nodes across two racks (two nodes/rack). ```text ------------------- | router | ------------------- | | --------------- --------------- | rack-0 | | rack-1 | --------------- --------------- | kube-node-0 | | kube-node-2 | - - - - - - - - - - - - - - - - | kube-node-1 | | kube-node-3 | - - - - - - - - - - - - - - - - ``` Using the pod IP range `192.168.0.0/16`, we target the following setup: reserve the `192.168.0.0/24` and `192.168.1.0/24` pools for `rack-0`, `rack-1`. Let's get started. By installing Calico Cloud without setting the default IP pool to match, running `calicoctl get ippool -o wide` shows that Calico Cloud created its default IP pool of `192.168.0.0/16`: ```text NAME CIDR NAT IPIPMODE DISABLED SELECTOR default-ipv4-ippool 192.168.0.0/16 true Always false all() ``` 1. Delete the default IP pool. Since the `default-ipv4-ippool` IP pool resource already exists and accounts for the entire `/16` block, we will have to delete this first: ```bash kubectl delete ippools default-ipv4-ippool ``` 2. Label the nodes. To assign IP pools to specific nodes, these nodes must be labelled using [kubectl label](https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes/#add-a-label-to-a-node). ```bash kubectl label nodes kube-node-0 rack=0 kubectl label nodes kube-node-1 rack=0 kubectl label nodes kube-node-2 rack=1 kubectl label nodes kube-node-3 rack=1 ``` 3. Create an IP pool for each rack. ```bash kubectl create -f -< nginx-5c7588df-s7qw6 1/1 Running 0 6m7s 192.168.0.129 kube-node-1 nginx-5c7588df-w7r7g 1/1 Running 0 6m3s 192.168.1.65 kube-node-2 nginx-5c7588df-62lnf 1/1 Running 0 6m3s 192.168.1.1 kube-node-3 nginx-5c7588df-pnsvv 1/1 Running 0 6m3s 192.168.1.64 kube-node-2 ``` The grouping of IP addresses assigned to the workloads differ based on what node that they were scheduled to. Additionally, the assigned address for each workload falls within the respective IP pool that selects the rack that they run on. > **SECONDARY:** Calico Cloud IPAM will not reassign IP addresses to workloads that are already running. To update running workloads with IP addresses from a newly configured IP pool, they must be recreated. We recommend doing this before going into production or during a maintenance window. ## Additional resources[​](#additional-resources) [Calico IPAM](https://docs.tigera.io/calico-cloud/reference/component-resources/configuration) ### Migrate from one IP pool to another ## Big picture[​](#big-picture) Migrate pods from one IP pool to another on a running cluster without network disruption. ## Value[​](#value) Pods are assigned IP addresses from IP pools that you configure in Calico Cloud. As the number of pods increase, you may need to increase the number of addresses available for pods to use. Or, you may need to move pods from a CIDR that was used by mistake. Calico Cloud lets you migrate from one IP pool to another one on a running cluster without network disruption. ## Concepts[​](#concepts) ### IP pools and cluster CIDRs[​](#ip-pools-and-cluster-cidrs) Calico Cloud supports using multiple disjoint IP pool CIDRs within the cluster. However, Kubernetes expects that all pods have addresses within the same cluster CIDR. This means that although it is technically feasible to create an IP pool outside of the cluster CIDR, we do not recommend it. Pods allocated addresses outside of the Kubernetes cluster CIDR will lose network connectivity. ## Before you begin...[​](#before-you-begin) **Verify that you are using Calico Cloud IPAM**. If you are not sure which IPAM your cluster is using, the way to tell depends on install method. The IPAM plugin can be queried on the default Installation resource. ```text kubectl get installation default -o go-template --template {{.spec.cni.ipam.type}} ``` If your cluster is using Calico IPAM, the above command should return a result of `Calico`. **Verify orchestrator support for changing the pod network CIDR**. Although Kubernetes supports changing the pod network CIDR, not all orchestrators do. For example, OpenShift does not support this feature as described in ## How to[​](#how-to) ### Migrate from one IP pool to another[​](#migrate-from-one-ip-pool-to-another-1) Follow these steps to migrate pods from one IP pool to another pool. > **SECONDARY:** If you follow these steps, existing pod connectivity will not be affected. (If you delete the old IP pool before you create and verify the new pool, existing pods will be affected.) When pods are deleted, applications may be temporarily unavailable (depending on the type of application); plan accordingly. 1. Add a new IP pool. > **SECONDARY:** It is highly recommended that your Calico IP pools are within the Kubernetes cluster CIDR. If pods IPs are allocated from outside of the Kubernetes cluster CIDR, some traffic flows may have NAT applied unnecessarily causing unexpected behavior. > > If you have configured kube-proxy with a cluster CIDR, you must ensure that it contains the new IP pool CIDR in order to prevent the unwanted NAT. 2. Disable the old IP pool. > **SECONDARY:** Disabling an IP pool only prevents new IP address allocations; it does not affect the networking of existing pods. 3. Delete pods from the old IP pool. This includes any new pods that may have been created with the old IP pool prior to disabling the pool. 4. Verify that new pods get an address from the new IP pool. 5. Delete the old IP pool. ## Tutorial[​](#tutorial) In the following example, we created a Kubernetes cluster using **kubeadm**. But the IP pool CIDR we configured (192.168.0.0/16) doesn't match the Kubernetes cluster CIDR. Let's change the CIDR to **10.0.0.0/16**, which for the purposes of this example falls within the cluster CIDR. Let’s run `calicoctl get ippool -o wide` to see the IP pool, **default-ipv4-ippool**. ```text NAME CIDR NAT IPIPMODE VXLANMODE DISABLED default-ipv4-ippool 192.168.0.0/16 true Always Never false ``` When we run `calicoctl get wep --all-namespaces`, we see that a pod is created using the default range (192.168.52.130/32). ```text NAMESPACE WORKLOAD NODE NETWORKS INTERFACE kube-system coredns-6f4fd4bdf-8q7zp vagrant 192.168.52.130/32 cali800a63073ed ``` Let’s get started changing this pod to the new IP pool (10.0.0.0/16). ### Step 1: Add a new IP pool[​](#step-1-add-a-new-ip-pool) We add a new **IPPool** with the CIDR range, **10.0.0.0/16**. ```yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: new-pool spec: cidr: 10.0.0.0/16 ipipMode: Always natOutgoing: true ``` Let’s verify the new IP pool. ```bash calicoctl get ippool -o wide ``` ```text NAME CIDR NAT IPIPMODE DISABLED default-ipv4-ippool 192.168.0.0/16 true Always false new-pool 10.0.0.0/16 true Always false ``` ### Step 2: Disable the old IP pool[​](#step-2-disable-the-old-ip-pool) List the existing IP pool definition. ```bash calicoctl get ippool -o yaml > pools.yaml ``` ```yaml apiVersion: projectcalico.org/v3 items: - apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: default-ipv4-ippool spec: cidr: 192.0.0.0/16 ipipMode: Always natOutgoing: true - apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: new-pool spec: cidr: 10.0.0.0/16 ipipMode: Always natOutgoing: true ``` Edit pools.yaml. Disable this IP pool by setting: `disabled: true` ```yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: default-ipv4-ippool spec: cidr: 192.0.0.0/16 ipipMode: Always natOutgoing: true disabled: true ``` Apply the changes. Remember, disabling a pool only affects new IP allocations; networking for existing pods is not affected. ```bash kubectl apply -f pools.yaml ``` Verify the changes. ```bash calicoctl get ippool -o wide ``` ```text NAME CIDR NAT IPIPMODE DISABLED default-ipv4-ippool 192.168.0.0/16 true Always true new-pool 10.0.0.0/16 true Always false ``` ### Step 3: Delete pods from the old IP pool[​](#step-3-delete-pods-from-the-old-ip-pool) Next, we delete all of the existing pods from the old IP pool. (In our example, **coredns** is our only pod; for multiple pods you would trigger a deletion for all pods in the cluster.) ```bash kubectl delete pod -n kube-system coredns-6f4fd4bdf-8q7zp ``` ### Step 4: Verify that new pods get an address from the new IP pool[​](#step-4-verify-that-new-pods-get-an-address-from-the-new-ip-pool) 1. Create a test namespace and nginx pod. ```bash kubectl create ns ippool-test ``` 2. Create an nginx pod. ```bash kubectl -n ippool-test create deployment nginx --image nginx ``` 3. Verify that the new pod gets an IP address from the new range. ```bash kubectl -n ippool-test get pods -l app=nginx -o wide ``` 4. Clean up the ippool-test namespace. ```bash kubectl delete ns ippool-test ``` ### Step 5: Delete the old IP pool[​](#step-5-delete-the-old-ip-pool) Now that you've verified that pods are getting IPs from the new range, you can safely delete the old pool. ```bash kubectl delete ippool default-ipv4-ippool ``` ## Additional resources[​](#additional-resources) - [IP pools reference](https://docs.tigera.io/calico-cloud/reference/resources/ippool) ### Change IP pool block size ## Big picture[​](#big-picture) Change the IP pool block size to efficiently manage IP pool addresses. ## Value[​](#value) Changing IP pool block size after installation requires ordered steps to minimize pod connectivity disruption. ## Concepts[​](#concepts) ### About IP pools[​](#about-ip-pools) By default, Calico Cloud uses an IPAM block size of 64 addresses – /26 for IPv4, and /122 for IPv6. However, the block size can be changed depending on the IP pool address family. - IPv4: 20-32, inclusive - IPv6: 116-128, inclusive You can have **only one default IP pool for per protocol** in your installation manifest. In this example, there is one IP pool for IPv4 (/26), and one IP pool for IPv6 (/122). ```yaml apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: # Configures Calico networking. calicoNetwork: ipPools: - blockSize: 26 cidr: 10.48.0.0/21 encapsulation: IPIP natOutgoing: Enabled nodeSelector: all() - blockSize: 122 cidr: 2001::00/64 encapsulation: None natOutgoing: Enabled nodeSelector: all() ``` However, the following is invalid because it has two IP pools for IPv4. ```yaml apiVersion: operator.tigera.io/v1 kind: Installation metadata: name: default spec: # Configures Calico networking. calicoNetwork: ipPools: - blockSize: 26 cidr: 10.48.0.0/21 encapsulation: IPIP natOutgoing: Enabled nodeSelector: all() - blockSize: 31 cidr: 10.48.8.0/21 encapsulation: IPIP natOutgoing: Enabled nodeSelector: all() ``` ### Expand or shrink IP pool block sizes[​](#expand-or-shrink-ip-pool-block-sizes) By default, the Calico Cloud IPAM block size for an IP pool is /26. To expand from the default size /26, lower the `blockSize` (for example, /24). To shrink the `blockSize` from the default /26, raise the number (for example, /28). ### Best practice: change IP pool block size before installation[​](#best-practice-change-ip-pool-block-size-before-installation) Because the `blockSize` field cannot be edited directly after Calico Cloud installation, it is best to change the IP pool block size before installation to minimize disruptions to pod connectivity. ## Before you begin...[​](#before-you-begin) **Required** Verify that you are using Calico Cloud IPAM. If you are not sure which IPAM your cluster is using, the way to tell depends on install method. The IPAM plugin can be queried on the default Installation resource. ```text kubectl get installation default -o go-template --template {{.spec.cni.ipam.type}} ``` If your cluster is using Calico IPAM, the above command should return a result of `Calico`. ## How to[​](#how-to) > **SECONDARY:** Follow the steps to minimize pod connectivity disruption. Pods may lose connectivity when they are redeployed, and may lose external connectivity while in the temporary pool. Also, when pods are deleted, applications may be temporarily unavailable (depending on the type of application). Plan your changes accordingly. The high-level steps to follow are: 1. [Create a temporary IP pool](#create-a-temporary-ip-pool) **Note**: The temporary IP pool must not overlap with the existing one. 2. [Disable the existing IP pool](#disable-the-existing-ip-pool) **Note**: When you disable an IP pool, only new IP address allocations are prevented; networking of existing pods are not affected. 3. [Delete pods from the existing IP pool](#delete-pods-from-the-existing-ip-pool) This includes any new pods that may have been created with the existing IP pool prior to disabling the pool. Verify that new pods get an address from the temporary IP pool. 4. [Delete the existing IP pool](#delete-the-existing-ip-pool) 5. [Create a new IP pool with the desired block size](#create-a-new-ip-pool-with-the-desired-block-size) 6. [Disable the temporary IP pool](#disable-the-temporary-ip-pool) 7. [Delete pods from the temporary IP pool](#delete-pods-from-the-temporary-ip-pool) 8. [Delete the temporary IP pool](#delete-the-temporary-ip-pool) ## Tutorial[​](#tutorial) In the following steps, our Kubernetes cluster has a default CIDR block size of /26. We want to shrink the block size to /28 to use the pool more efficiently. ### Create a temporary IP pool[​](#create-a-temporary-ip-pool) We add a new IPPool with the CIDR range, 10.0.0.0/16. Create a temporary-pool.yaml. ```yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: temporary-pool spec: cidr: 10.0.0.0/16 ipipMode: Always natOutgoing: true ``` Apply the changes. ```bash kubectl apply -f temporary-pool.yaml ``` Let’s verify the temporary IP pool. ```bash calicoctl get ippool -o wide ``` ```text NAME CIDR NAT IPIPMODE DISABLED default-ipv4-ippool 192.168.0.0/16 true Always false temporary-pool 10.0.0.0/16 true Always false ``` ### Disable the existing IP pool[​](#disable-the-existing-ip-pool) Disable allocations in the default pool. ```bash kubectl patch ippool default-ipv4-ippool -p '{"spec": {"disabled": "true"}}' ``` Verify the changes. ```bash calicoctl get ippool -o wide ``` ```text NAME CIDR NAT IPIPMODE DISABLED default-ipv4-ippool 192.168.0.0/16 true Always true temporary-pool 10.0.0.0/16 true Always false ``` ### Delete pods from the existing IP pool[​](#delete-pods-from-the-existing-ip-pool) In our example, **coredns** is our only pod; for multiple pods you would trigger a deletion for all pods in the cluster. ```bash kubectl delete pod -n kube-system coredns-6f4fd4bdf-8q7zp ``` Restart all pods with just one command. > **WARNING:** The following command is disruptive and may take several minutes depending on the number of pods deployed. ```bash kubectl delete pod -A --all ``` ### Delete the existing IP pool[​](#delete-the-existing-ip-pool) Now that you’ve verified that pods are getting IPs from the new range, you can safely delete the existing pool. ```bash kubectl delete ippool default-ipv4-ippool ``` ### Create a new IP pool with the desired block size[​](#create-a-new-ip-pool-with-the-desired-block-size) In this step, we update the IPPool with the new block size of (/28). ```yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: default-ipv4-ippool spec: blockSize: 28 cidr: 192.0.0.0/16 ipipMode: Always natOutgoing: true ``` Apply the changes. ```bash kubectl apply -f pool.yaml ``` ### Disable the temporary IP pool[​](#disable-the-temporary-ip-pool) ```bash kubectl patch ippool temporary-pool -p '{"spec": {"disabled": "true"}}' ``` ### Delete pods from the temporary IP pool[​](#delete-pods-from-the-temporary-ip-pool) In our example, **coredns** is our only pod; for multiple pods you would trigger a deletion for all pods in the cluster. ```bash kubectl delete pod -n kube-system coredns-6f4fd4bdf-8q7zp ``` Restart all pods with just one command. > **WARNING:** The following command is disruptive and may take several minutes depending on the number of pods deployed. ```bash kubectl delete pod -A --all ``` Validate your pods and block size are correct by running the following commands: ```bash kubectl get pods --all-namespaces -o wide calicoctl ipam show --show-blocks ``` ### Delete the temporary IP pool[​](#delete-the-temporary-ip-pool) Clean up the IP pools by deleting the temporary IP pool. ```bash kubectl delete pool temporary-pool ``` ### Restrict a pod to use an IP address in a specific range ## Big picture[​](#big-picture) Restrict the IP address chosen for a pod to a specific range of IP addresses. ## Value[​](#value) When Kubernetes pods interact with external systems that make decisions based on IP ranges (for example legacy firewalls), it can be useful to define several IP ranges and explicitly assign pods to those ranges. Using Calico Cloud IP Address Management (IPAM), you can restrict a pod to use an address from within a specific range. ## Concepts[​](#concepts) ### Kubernetes pod CIDR[​](#kubernetes-pod-cidr) The **Kubernetes pod CIDR** is the range of IPs Kubernetes expects pod IPs to be assigned from. It is defined for the entire cluster and is used by various Kubernetes components to determine whether an IP belongs to a pod. For example, kube-proxy treats traffic differently if that traffic is from a pod than if it is not. All pod IPs must be in the CIDR range for Kubernetes to function correctly. ### IP Pool[​](#ip-pool) **IP pools** are ranges of IP addresses from which Calico Cloud assigns pod IPs. By default, Calico Cloud creates an IP pool for the entire Kubernetes pod CIDR, but you can change this to break the pod CIDR up into several pools. You can control which pool Calico Cloud uses for each pod using node selectors, or annotations on the pod or the pod’s namespace. ## Before you begin...[​](#before-you-begin) The features in this How to guide require: - Calico Cloud IPAM If you are not sure which IPAM your cluster is using, the way to tell depends on install method. The IPAM plugin can be queried on the default Installation resource. ```text kubectl get installation default -o go-template --template {{.spec.cni.ipam.type}} ``` If your cluster is using Calico IPAM, the above command should return a result of `Calico`. Additionally, cluster administrators must have [configured IP pools](https://docs.tigera.io/calico-cloud/reference/resources/ippool) to define the valid IP ranges to use for allocating pod IP addresses. ## How to[​](#how-to) ### Restrict a pod to use an IP address range[​](#restrict-a-pod-to-use-an-ip-address-range) Annotate the pod with key `cni.projectcalico.org/ipv4pools` and/or `cni.projectcalico.org/ipv6pools` and value set to a list of IP pool names, enclosed in brackets. For example: ```text cni.projectcalico.org/ipv4pools: '["pool-1", "pool-2"]' ``` Note the use of the escaped " for the inner double quotes around the pool names. ### Restrict all pods within a namespace to use an IP address range[​](#restrict-all-pods-within-a-namespace-to-use-an-ip-address-range) Annotate the namespace with key `cni.projectcalico.org/ipv4pools` and/or `cni.projectcalico.org/ipv6pools` and value set to a list of IP pool names, enclosed in brackets. For example: ```text cni.projectcalico.org/ipv4pools: '["pool-1", "pool-2"]' ``` Note the use of the escaped `\"` for the inner double quotes around the pool names. If both the pod and the pod’s namespace have the annotation, the pod annotation takes precedence. The annotation must be present at the time the pod is created. Adding it to an existing pod has no effect. ## Additional resources[​](#additional-resources) For help configuring Calico Cloud IPAM, see [Configuring the Calico Cloud CNI Plugins](https://docs.tigera.io/calico-cloud/reference/component-resources/configuration). ### LoadBalancer IP address management ## Understanding Calico Cloud LoadBalancer IP address management[​](#understanding-calico-cloud-loadbalancer-ip-address-management) You might want to utilize Service of type LoadBalancer in your cluster to provide stable long-lasting IP address to access your deployed application. Calico Cloud can help you with providing and managing IP addresses to your Services in your cluster. Calico Cloud comes with LoadBalancer IPAM deployed as part of Calico Cloud Kube-controllers. ### Before you begin...[​](#before-you-begin) Ensure that you have a cluster with Calico Cloud installed, and kube-controllers running. ### Enable LoadBalancer controller[​](#enable-loadbalancer-controller) By default, LoadBalancer controller is enabled in Calico Cloud kube-controllers configuration. You can verify the status by checking the KubeControllersConfiguration resource. ```bash kubectl get kubecontrollersconfiguration default -o yaml ``` The output should contain LoadBalancer controller configuration as follows: ```yaml spec: controllers: loadBalancer: assignIPs: AllServices ``` If LoadBalancer controller is not enabled, you can enable it by patching the KubeControllersConfiguration resource: ```bash kubectl patch kubecontrollersconfiguration default --type=merge --patch '{"spec": {"controllers":{"loadBalancer":{"assignIPs": "AllServices"}}}}' ``` ### IP Pool for Service of type LoadBalancer[​](#ip-pool-for-service-of-type-loadbalancer) Calico Cloud does not automatically provide default IP Pool for LoadBalancer IP address management. You will need to create an IP Pool with `allowedUses` LoadBalancer for Calico Cloud to start assigning Service IPs. ```yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: loadbalancer-ip-pool spec: cidr: 192.210.0.0/20 blockSize: 24 natOutgoing: true disabled: false assignmentMode: Automatic allowedUses: - LoadBalancer ``` > **SECONDARY:** You can create multiple IP Pools with allowedUses LoadBalancer as long as there are no CIDR conflicts. By setting assignmentMode to Manual you can reserve the IP Pool for manual assignments only. Explore more about [manual assignment](#manual-service-ip-address-assignment) ### Automatic Service IP address assignment[​](#automatic-service-ip-address-assignment) When you create Service of type LoadBalancer Calico Cloud kube-controller will automatically detect the new Service and assign an IP address from available IP Pool. If no address is available, the Service will remain in pending state until an address is available. ```yaml apiVersion: v1 kind: Service metadata: name: service-loadbalancer spec: selector: app: nginx ports: - port: 80 targetPort: 80 name: default type: LoadBalancer ``` ### Manual Service IP address assignment[​](#manual-service-ip-address-assignment) There are cases where you would like to be more specific about what IP address Calico Cloud assigns to your Service. With annotations, you can specify how IP address should be assigned. The annotations can be added and removed as needed during the lifetime of the Service. When you remove an annotation Calico Cloud kube-controller will check if the assigned IP is still valid and potentially assign a new one. #### Specify IP Pool[​](#specify-ip-pool) ##### IPv4 Pool[​](#ipv4-pool) Annotate the Service with projectcalico.org/ipv4pools to assign IP address from the selected IP Pools. You can specify multiple IP Pools in this annotation, Calico Cloud will pick an available IP address to assign. ```text "projectcalico.org/ipv4pools": '["loadBalancerIPv4Pool"]' ``` ##### IPv6 Pool[​](#ipv6-pool) Annotate the Service with projectcalico.org/ipv6pools to assign IP address from the selected IP Pools. You can specify multiple IP Pools in this annotation, Calico Cloud will pick an available IP address to assign. ```text "projectcalico.org/ipv6pools": '["loadBalancerIPv6Pool"]' ``` > **SECONDARY:** In dual-stack environment you don't have to specify both annotations. If IP Pool was not specified for IP family, Calico Cloud will automatically assign available IP address from available IP Pool #### Specifying IP address[​](#specifying-ip-address) Annotate the Service with projectcalico.org/loadBalancerIPs to assign specific IP address. The address must be available otherwise Calico Cloud will not be able to assign the address. Currently you can only specify one IPv4 and one IPv6 address at the same time. ```text "projectcalico.org/loadBalancerIPs": '["x.x.x.x"]' ``` > **SECONDARY:** If Service contains "projectcalico.org/loadBalancerIPs" annotation and either "projectcalico.org/ipv6pools" or "projectcalico.org/ipv4pools" are present, Calico Cloud will favour projectcalico.org/loadBalancerIPs annotation and try to assign that IP address. There is no fall back to an IP Pool if the specified address is not available. ### Manage LoadBalancer kube-controller assignment mode[​](#manage-loadbalancer-kube-controller-assignment-mode) In certain cases you might not wish for Calico Cloud to automatically assign IP addresses to your Service. This can be useful in case you have multiple Service IPAM solutions in your cluster. LoadBalancer kube-controller is able to operate in two distinct modes `AllServices` in which Calico Cloud assigns IP address to each Service in your cluster and `RequestedServicesOnly` in which Calico Cloud only assigns IP addresses to Service with annotations mentioned above or if `Service.spec.loadBalancerClass` is set to `calico`. You can change the mode at any point, but note that switching to `RequestedServicesOnly` will unassign any addresses from Services that do not contain the above annotations. ```bash kubectl patch kubecontrollersconfiguration default --type=merge --patch '{"spec": {"controllers":{"loadBalancer":{"assignIPs": "RequestedServicesOnly"}}}}' ``` > **SECONDARY:** LoadBalancer controller considers the `Service.spec.loadBalancerClass`, it will skip assignment for Services where `loadBalancerClass` is set to a value other than `calico`, regardless of the `assignIPs` setting. ### Additional resources[​](#additional-resources) Calico LoadBalancer IP address management works in conjunction with other Calico Cloud components. Calico Cloud kube-controllers provides only the IP address management, to advertise LoadBalancer IPs you will have to update your BGP configuration. You can find out more information at: - [Advertising Kubernetes service IP addresses](https://docs.tigera.io/calico-cloud/networking/configuring/advertise-service-ips#advertise-service-load-balancer-ip-addresses) - [IP Pool](https://docs.tigera.io/calico-cloud/reference/resources/ippool) - [Kube-controllers configuration](https://docs.tigera.io/calico-cloud/reference/resources/kubecontrollersconfig#loadbalancercontroller) --- ## Cluster mesh ### Cluster mesh With cluster mesh, you can secure cross-cluster connections using identity-aware network policy and federate services for cross-cluster service discovery. ##### [Overview](https://docs.tigera.io/calico-cloud/multicluster/overview) [Configure a cluster mesh for cross-cluster endpoints sharing, cross-cluster connectivity, and cross-cluster service discovery.](https://docs.tigera.io/calico-cloud/multicluster/overview) ##### [Configure federated services](https://docs.tigera.io/calico-cloud/multicluster/services-controller) [Configure a federated service for cross-cluster service discovery for local clusters.](https://docs.tigera.io/calico-cloud/multicluster/services-controller) ##### [Cluster mesh example for clusters in AWS](https://docs.tigera.io/calico-cloud/multicluster/aws) [A sample configuration of Calico Enterprise federated endpoint identity and federated services for an AWS cluster.](https://docs.tigera.io/calico-cloud/multicluster/aws) ##### [Creating the cluster mesh](https://docs.tigera.io/calico-cloud/multicluster/kubeconfig) [Configure a local cluster to pull endpoint data from a remote cluster.](https://docs.tigera.io/calico-cloud/multicluster/kubeconfig) ### Overview ## Big picture[​](#big-picture) Secure cross-cluster connections with identity-aware network policy, and federate services for cross-cluster service discovery. Utilize Calico Cloud to establish cross-cluster connectivity. ## Value[​](#value) At some point in your Kubernetes journey, you may have applications that need to access services and workloads running in another cluster. By default, pods can only communicate with pods within the same cluster. Additionally, services and network policy only select pods from within the same cluster. Calico Cloud can help overcome these barriers by forming a cluster mesh the following features: - **Federated endpoint identity** Allow a local Kubernetes cluster to include the workload endpoints (pods) and host endpoints of a remote cluster in the calculation of local network policies applied on each node of the local cluster. - **Federated services** Enable a local Kubernetes Service to populate with Endpoints selected from both local cluster and remote cluster Services. - **Multi-cluster networking** Establish an overlay network between clusters to provide cross-cluster connectivity with Calico Cloud. ## Concepts[​](#concepts) ### Pod IP routability[​](#pod-ip-routability) Calico Cloud cluster mesh is implemented at Kubernetes at the network layer, based on pod IPs. Taking advantage of federated workload endpoint identity and federated services requires that pod IPs are routable between clusters. This is because identity-aware network policy requires source and destination pod IPs to be preserved to establish pod identity. Additionally, the Endpoint IPs of pods selected by a federated Service must be routable in order for that Service to be of value. You can utilize Calico Cloud multi-cluster networking to establish pod IP routability between clusters via overlay. Alternatively, you can manually set up pod IP routability between clusters without encapsulation (e.g. VPC routing, BGP routing). ### Federated endpoint identity[​](#federated-endpoint-identity) Federated endpoint identity in a cluster mesh allows a local Kubernetes cluster to include the workload endpoints (pods) and host endpoints of a remote cluster in the calculation of the local policies for each node, e.g. Cluster A network policy allows its application pods to talk to database pods in Cluster B. This feature does not *federate network policies*; policies from a remote cluster are not applied to the endpoints on the local cluster, and the policy from the local cluster is rendered only locally and applied to the local endpoints. ### Federated services[​](#federated-services) Federated services in a cluster mesh works with federated endpoint identity, providing cross-cluster service discovery for a local cluster. If you have an existing service discovery mechanism, this feature is optional. Federated services use the Tigera Federated Services Controller to federate all Kubernetes endpoints (workload and host endpoints) across all of the clusters. The Federated Services Controller accesses service and endpoints data in the remote clusters directly through the Kubernetes API. ## Next steps[​](#next-steps) [Configure remote-aware policy and multi-cluster networking](https://docs.tigera.io/calico-cloud/multicluster/kubeconfig) ### Creating the cluster mesh In this page, we will create a Calico Cloud cluster mesh by connecting clusters together. Once created, Calico Cloud cluster mesh enables multi-cluster networking, network policy for cross-cluster connections, cross-cluster services, and encryption via WireGuard. ## Requirements[​](#requirements) Calico Cloud multi-cluster networking provides routing between clusters that preserves pod IPs. This section outlines the requirements for this routing to be established. If your network already provides routing between clusters that preserves pod IPs, you can skip this section. ### Prerequisites for Calico Cloud multi-cluster networking:[​](#prerequisites-for-calico-cloud-multi-cluster-networking) - All nodes participating in the cluster mesh must be able to establish connections to each other via their private IP. - All nodes participating in the cluster mesh must have unique node names. - Pod CIDRs between clusters must not overlap. - All clusters must have at least one overlay network in common (VXLAN and/or WireGuard). - All clusters must have the same `routeSource` setting on `FelixConfiguration`. If using VXLAN: - The `vxlan*` settings on `FelixConfiguration` must be the same across clusters participating in the mesh. - The underlying network must allow traffic on `vxlanPort` between clusters participating in the mesh. - All clusters must use Calico CNI. If using WireGuard: - The `wireguard*` settings on `FelixConfiguration` must be the same across clusters participating in the mesh. - The underlying network must allow traffic on `wireguardListeningPort` between clusters participating in the mesh. - All clusters must use Calico CNI OR All clusters must use non-Calico CNI (mixing non-Calico CNI types is supported). Note: much like intra-cluster routing in Calico Cloud, cross-cluster routing can utilize both VXLAN and WireGuard at the same time. If both are enabled and a WireGuard peer is not ready, communication with that peer will fall back to VXLAN. ## Setup[​](#setup) ### Generate credentials for cross-cluster resource synchronization[​](#generate-credentials-for-cross-cluster-resource-synchronization) > **SUCCESS:** The basis of cluster mesh is the ability for a cluster connect to a remote cluster and sync data from it. This enables each Calico Cloud cluster to have a view into the datastore that includes both local and remote cluster pods. In this section, we will create a `kubeconfig` for each cluster. This `kubeconfig` is what other clusters will use to connect to a given cluster and synchronize data from it. **For each** cluster in the cluster mesh, utilizing an existing `kubeconfig` with administrative privileges, follow these steps: 1. Create the ServiceAccount used by remote clusters for authentication: ```bash kubectl apply -f https://downloads.tigera.io/ee/v3.22.0-2.0/manifests/federation-remote-sa.yaml ``` 2. Create the ClusterRole and ClusterRoleBinding used by remote clusters for authorization: ```bash kubectl apply -f https://downloads.tigera.io/ee/v3.22.0-2.0/manifests/federation-rem-rbac-kdd.yaml ``` 3. Create the ServiceAccount token that will be used in the `kubeconfig`: ```yaml kubectl apply -f - < $KUBECONFIG_NAME ``` 5. Verify that the `kubeconfig` file works: Issue the following command to validate the `kubeconfig` file can be used to connect to the current cluster and access resources: ```bash kubectl --kubeconfig=$KUBECONFIG_NAME get nodes ``` Once you've created a `kubeconfig` for **each** cluster, you can proceed to the next section to establish the cluster connections that form the mesh. ### Establish cross-cluster resource synchronization[​](#establish-cross-cluster-resource-synchronization) > **SUCCESS:** The cluster mesh is formed when each cluster connects to every other cluster to synchronize data. A cluster connects to another cluster using a RemoteClusterConfiguration, which references a kubeconfig created for the remote cluster. In this section, within each cluster, we will create a RemoteClusterConfiguration for each other cluster in the mesh. This RemoteClusterConfiguration instructs the cluster to connect to a cluster using a kubeconfig. With each cluster being connected to each other cluster, a full cluster mesh will be formed. **Tab: Calico Cloud multi-cluster routing** > **SUCCESS:** Calico Cloud achieves cross-cluster routing by extending the overlay network of a cluster to include nodes from remote clusters. This is made possible by each cluster having a view into the datastore that now includes remote pods and nodes. **For each pair** of clusters in the cluster mesh (e.g. {A,B}, {A,C}, {B,C} for clusters A,B,C): 1. In cluster 1, create a secret that contains the `kubeconfig` for cluster 2: Determine the namespace (``) for the secret to replace in all steps. The simplest method to create a secret for a remote cluster is to use the `kubectl` command because it correctly encodes the data and formats the file. ```bash kubectl create secret generic remote-cluster-secret-name -n \ --from-literal=datastoreType=kubernetes \ --from-file=kubeconfig= ``` 2. If RBAC is enabled in cluster 1, create a Role and RoleBinding for Calico Cloud to use to access the secret that contains the `kubeconfig` for cluster 2: ```bash kubectl create -f - < rules: - apiGroups: [""] resources: ["secrets"] verbs: ["watch", "list", "get"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: remote-cluster-secret-access namespace: roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: remote-cluster-secret-access subjects: - kind: ServiceAccount name: calico-typha namespace: calico-system EOF ``` 3. Create the RemoteClusterConfiguration in cluster 1: Within the RemoteClusterConfiguration, we specify the secret used to access cluster 2, and the overlay routing mode which toggles the establishment of cross-cluster overlay routes. ```bash kubectl create -f - < kind: Secret syncOptions: overlayRoutingMode: Enabled EOF ``` 4. [Validate](#check-remote-cluster-connection) the that the remote cluster connection can be established. 5. Repeat the above steps, switching cluster 1 and cluster 2. **Tab: Network-provided routing** In this setup, the cluster mesh will rely on the underlying network to provides cross-cluster routing that preserves pod IPs. **For each pair** of clusters in the cluster mesh (e.g. {A,B}, {A,C}, {B,C} for clusters A,B,C): 1. In cluster 1, create a secret that contains the `kubeconfig` for cluster 2: Determine the namespace (``) for the secret to replace in all steps. The simplest method to create a secret for a remote cluster is to use the `kubectl` command because it correctly encodes the data and formats the file. ```bash kubectl create secret generic remote-cluster-secret-name -n \ --from-literal=datastoreType=kubernetes \ --from-file=kubeconfig= ``` 2. If RBAC is enabled in cluster 1, create a Role and RoleBinding for Calico Cloud to use to access the secret that contains the `kubeconfig` for cluster 2: ```bash kubectl create -f - < rules: - apiGroups: [""] resources: ["secrets"] verbs: ["watch", "list", "get"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: remote-cluster-secret-access namespace: roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: remote-cluster-secret-access subjects: - kind: ServiceAccount name: calico-typha namespace: calico-system EOF ``` 3. Create the RemoteClusterConfiguration in cluster 1: Within the RemoteClusterConfiguration, we specify the secret used to access cluster 2, and the overlay routing mode which toggles the establishment of cross-cluster overlay routes. ```bash kubectl create -f - < kind: Secret syncOptions: overlayRoutingMode: Disabled EOF ``` 4. If you have no IP pools in cluster 1 with NAT-outgoing enabled, skip this step. Otherwise, if you have IP pools in cluster 1 with NAT-outgoing enabled, and workloads in that pool will egress to workloads in cluster 2, you need to instruct Calico Cloud to not perform NAT on traffic destined for IP pools in cluster 2. You can achieve this by creating a disabled IP pool in cluster 1 for each CIDR in cluster 2. This IP pool should have NAT-outgoing disabled. For example: ```yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: cluster2-main-pool spec: cidr: disabled: true ``` 5. [Validate](#check-remote-cluster-connection) the that the remote cluster connection can be established. 6. Repeat the above steps, switching cluster 1 and cluster 2. #### 🎉 Done\![​](#-done) After completing the above steps for all cluster pairs in the cluster mesh, your clusters should now be forming a cluster mesh! You should now be able to route traffic between clusters, and write policy that can select remote workloads. > **SUCCESS:** A cluster in the mesh can write policy rules that select pods from other clusters in the mesh. This is because traffic from remote clusters has pod IPs preserved, and the local cluster can associate remote pod IPs with the pod specs it synchronized from remote clusters. ## How to[​](#how-to) ### Switch to multi-cluster networking[​](#switch-to-multi-cluster-networking) The steps above assume that you are configuring both federated endpoint identity and multi-cluster networking for the first time. If you already have federated endpoint identity, and want to use multi-cluster networking, follow these steps: 1. Validate that all [requirements](#calico-enterprise-multi-cluster-networking) for multi-cluster networking have been met. 2. Update the ClusterRole in each cluster in the cluster mesh using the RBAC manifest found in [Generate credentials for cross-cluster authentication](#generate-credentials-for-cross-cluster-resource-synchronization) 3. In all RemoteClusterConfigurations, set `Spec.OverlayRoutingMode` to `Enabled`. 4. Verify that all RemoteClusterConfigurations are bidirectional (in both directions for each cluster pair) using these [instructions](#establish-cross-cluster-resource-synchronization). 5. If you had previously created disabled IP pools to prevent NAT outgoing from applying to remote cluster destinations, those disabled IP pools are no longer needed when using multi-cluster networking and must be deleted. ### Validate federated endpoint identity & multi-cluster networking[​](#validate-federated-endpoint-identity--multi-cluster-networking) #### Validate RemoteClusterConfiguration and federated endpoint identity[​](#validate-remoteclusterconfiguration-and-federated-endpoint-identity) ##### Check remote cluster connection[​](#check-remote-cluster-connection) You can check the Typha logs for remote cluster connection status. Run the following command: ```bash kubectl logs deployment/calico-typha -n calico-system | grep "Sending in-sync update" ``` You should see an entry for each RemoteClusterConfiguration in the local cluster. If either output contains unexpected results, proceed to the [troubleshooting](#troubleshoot) section. #### Validate multi-cluster networking[​](#validate-multi-cluster-networking) If all requirements were met for Calico Cloud to establish multi-cluster networking, you can test the functionality by establishing a connection from a pod in a local cluster to the IP of a pod in a remote cluster. Ensure that there is no policy in either cluster that may block this connection. If the connection fails, proceed to the [troubleshooting](#troubleshoot) section. ### Create remote-identity-aware network policy[​](#create-remote-identity-aware-network-policy) With federated endpoint identity and routing between clusters established, you can now use labels to reference endpoints on a remote cluster in local policy rules, rather than referencing them by IP address. The main policy selector still refers only to local endpoints; and that selector chooses which local endpoints to apply the policy. However, rule selectors can now refer to both local and remote endpoints. In the following example, cluster A (an application cluster) has a network policy that governs outbound connections to cluster B (a database cluster). ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: default.app-to-db namespace: myapp spec: # The main policy selector selects endpoints from the local cluster only. selector: app == 'backend-app' tier: default egress: - destination: # Rule selectors can select endpoints from local AND remote clusters. selector: app == 'postgres' protocol: TCP ports: [5432] action: Allow ``` ### Troubleshoot[​](#troubleshoot) #### Troubleshoot RemoteClusterConfiguration and federated endpoint identity[​](#troubleshoot-remoteclusterconfiguration-and-federated-endpoint-identity) ##### Verify configuration[​](#verify-configuration) For each impacted remote cluster pair (between cluster A and cluster B): 1. Retrieve the `kubeconfig` from the secret stored in cluster A. Manually verify that it can be used to connect to cluster B. ```bash kubectl get secret -n remote-cluster-secret-name -o=jsonpath="{.data.kubeconfig}" | base64 -d > verify_kubeconfig_b kubectl --kubeconfig=verify_kubeconfig_b get nodes ``` This validates that the credentials used by Typha to connect to cluster B's API server are stored in the correct location and provide sufficient access. The command above should yield a result like the following: ```text NAME STATUS ROLES AGE VERSION clusterB-master Ready master 7d v1.27.0 clusterB-worker-1 Ready worker 7d v1.27.0 clusterB-worker-2 Ready worker 7d v1.27.0 ``` If you do not see the nodes of cluster B listed in response to the command above, verify that you [created](#generate-credentials-for-cross-cluster-resource-synchronization) the `kubeconfig` for cluster B correctly, and that you [stored](#establish-cross-cluster-resource-synchronization) it in cluster A correctly. If you do see the nodes of cluster B listed in response to the command above, you can run this test (or a similar test) on a node in cluster A to verify that cluster A nodes can connect to the API server of cluster B. 2. Validate that the Typha service account in Cluster A is authorized to retrieve the `kubeconfig` secret for cluster B. ```bash kubectl auth can-i list secrets --namespace --as=system:serviceaccount:calico-system:calico-typha ``` This command should yield the following output: ```text yes ``` If the command does not return this output, verify that you correctly [configured RBAC](#establish-cross-cluster-resource-synchronization) in cluster A. 3. Repeat the above, switching cluster A to cluster B. ##### Check logs[​](#check-logs) Validate that querying Typha logs yield the expected result outlined in the [validation](#validate-federated-endpoint-identity--multi-cluster-networking) section. If the Typha logs do not yield the expected result, review the warning or error-related logs in `typha` or `calico-node` for insights. #### Troubleshoot multi-cluster networking[​](#troubleshoot-multi-cluster-networking) ##### Basic validation[​](#basic-validation) - Ensure that RemoteClusterConfiguration and federated endpoint identity are [functioning correctly](#validate-federated-endpoint-identity--multi-cluster-networking) - Verify that you have met the [prerequisites](#calico-enterprise-multi-cluster-networking) for multi-cluster networking - If you had previously set up RemoteClusterConfigurations without multi-cluster networking, and are upgrading to use the feature, review the [switching considerations](#switch-to-multi-cluster-networking) - Verify that traffic between clusters is not being denied by network policy ##### Check overlayRoutingMode[​](#check-overlayroutingmode) Ensure that `overlayRoutingMode` is set to `"Enabled"` on all RemoteClusterConfigurations. If overlay routing is successfully enabled, you can view the logs of a Typha instance using: ```bash kubectl logs deployment/calico-typha -n calico-system ``` You should see an output for each connected remote cluster that looks like this: ```text 18:49:35.394 [INFO][14] wrappedcallbacks.go 443: Creating syncer for RemoteClusterConfiguration(my-cluster) 18:49:35.394 [INFO][14] watchercache.go 186: Full resync is required ListRoot="/calico/ipam/v2/assignment/" 18:49:35.395 [INFO][14] watchercache.go 186: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/workloadendpoints" 18:49:35.396 [INFO][14] watchercache.go 186: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/hostendpoints" 18:49:35.396 [INFO][14] watchercache.go 186: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/profiles" 18:49:35.396 [INFO][14] watchercache.go 186: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/nodes" 18:49:35.397 [INFO][14] watchercache.go 186: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/ippools" ``` If you do not see the each of the resource types above, overlay routing was not successfully enabled in your cluster. Verify that you followed the [setup](#establish-cross-cluster-resource-synchronization) correctly for overlay routing, and that the cluster is using a version of Calico Cloud that supports multi-cluster networking. ###### Check logs[​](#check-logs-1) Warning or error logs in `typha` or `calico-node` may provide insight into where issues are occurring. ## Next steps[​](#next-steps) [Configure federated services](https://docs.tigera.io/calico-cloud/multicluster/services-controller) ### Configure federated services ## Big picture[​](#big-picture) Configure local clusters to discover services across multiple clusters. ## Value[​](#value) Use cluster mesh and federated services discovery along with federated endpoint identity to extend and automate endpoints sharing. (Optional if you have your own service discovery mechanism.) ## Concepts[​](#concepts) ### Federated services[​](#federated-services) A federated service (also called a backing service), is a set of services with consolidated endpoints. Calico Cloud discovers services across a cluster mesh (both local cluster and remote clusters) and creates a "federated service" on the local cluster that encompasses all of the individual services. Federated services are managed by the Tigera Federated Service Controller, which monitors and maintains endpoints for each locally-federated service. The controller does not change configuration on remote clusters. A federated service looks similar to a regular Kubernetes service, but instead of using a pod selector, it uses an annotation. For example: ```yaml apiVersion: v1 kind: Service metadata: name: my-app-federated namespace: default annotations: federation.tigera.io/serviceSelector: run == "my-app" spec: ports: - name: my-app-ui port: 8080 protocol: TCP type: ClusterIP ``` | Annotation | Description | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `federation.tigera.io/serviceSelector` | Required field that specifies the services used in the federated service. Format is a standard Calico Cloud selector (i.e. the same as Calico Cloud policy resources) and selects services based on their labels. The selector annotation selects services, not pods. Only services in the same namespace as the federated service are included. This implies namespace names across clusters are linked (this is a basic premise of federated endpoint identity). If the value is incorrectly specified, the service is not federated and endpoint data is removed from the service. View the warning logs in the controller for any issues processing this value. | **Syntax and rules** - Services that you specify in the federated service must be in the same namespace or they are ignored. A basic assumption of federated endpoint identity is that namespace names are linked across clusters. - If you specify a `spec.Selector` in a federated service, the service is not federated. - You cannot federate another federated service. If a service has a federated services annotation, it is not included as a backing service of another federated service. - The target port number in the federated service ports is not used. **Match services using a label** You can also match services using a label. The label is implicitly added to each service, but it does not appear in `kubectl` when viewing the service. | Label | Description | | ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `federation.tigera.io/remoteClusterName` | Label added to all remote services that correspond to the Remote Cluster Configuration name for the remote cluster. Use this label to restrict the clusters selected by the services. **Note**: The label is not added for services in the local cluster. | **About endpoints** - Do not manually create or manage endpoints resources; let the Tigera controller do all of the work. User updates to endpoint resources are ignored. - Endpoints are selected only when the service port name and protocol in the federated service matches the port name and protocol in the backing service. - Endpoint data configured in the federated service is slightly modified from the original data of the backing service. For backing services on remote clusters, the `targetRef.name` field in the federated service adds the ``. For example, `/`. ## Before you begin[​](#before-you-begin) **Required** - [Configure federated endpoint identity](https://docs.tigera.io/calico-cloud/multicluster/kubeconfig) ## How to[​](#how-to) - [Create service resources](#create-service-resources) - [Create a federated service](#create-a-federated-service) - [Access a federated service](#access-a-federated-service) ### Create service resources[​](#create-service-resources) On each cluster in the mesh that is providing a particular service, create your service resources as you normal would with the following requirements: - Services must all be in the same namespace. - Configure each service with a common label key and value to identify the common set of services across your clusters (for example, `run=my-app`). Kubernetes manages the service by populating the service endpoints from the pods that match the selector configured in the service spec. ### Configure a federated service[​](#configure-a-federated-service) 1. On a cluster that needs to access the federated set of pods that are running an application, create a service on that cluster leaving the `spec selector` blank. 2. Set the `federation.tigera.io/serviceSelector` annotation to be a Calico Cloud selector that selects the previously-configured services using the matching label match (for example, `run == "my-app"`). The Federated Services Controller manages this service, populating the service endpoints from all of the services that match the service selector configured in the annotation. ### Access a federated service[​](#access-a-federated-service) Any application can access the federated service using the local DNS name for that service. The simplest way to access a federated service is through its corresponding DNS name. By default, Kubernetes adds DNS entries to access a service locally. For a service called `my-svc` in the namespace `my-namespace`, the following DNS entry would be added to access the service within the local cluster: ```text my-svc.my-namespace.svc.cluster.local ``` DNS lookup for this name returns the fixed ClusterIP address assigned for the federated service. The ClusterIP is translated in iptables to one of the federated service endpoint IPs, and is load balanced across all of the endpoints. ## Tutorial[​](#tutorial) ### Create a service[​](#create-a-service) In the following example, the remote cluster defines the following service. ```yaml apiVersion: v1 kind: Service metadata: labels: run: my-app name: my-app namespace: default spec: selector: run: my-app ports: - name: my-app-ui port: 80 protocol: TCP targetPort: 9000 - name: my-app-console port: 81 protocol: TCP targetPort: 9001 type: ClusterIP ``` This service definition exposes two ports for the application `my-app`. One port for accessing a UI, and the other for accessing a management console. The service specifies a Kubernetes selector, which means the endpoints for this service are automatically populated by Kubernetes from matching pods within the services own cluster. ### Create a federated service[​](#create-a-federated-service) To create a federated service on your local cluster that federates the web access port for both the local and remote service, you would create a service resource on your local cluster as follows: ```yaml apiVersion: v1 kind: Service metadata: name: my-app-federated namespace: default annotations: federation.tigera.io/serviceSelector: run == "my-app" spec: ports: - name: my-app-ui port: 8080 protocol: TCP type: ClusterIP ``` The `spec.selector` is not specified so it will not be managed by Kubernetes. Instead, we use a `federation.tigera.io/selector` annotation, indicating that this is a federated service managed by the Federated Services Controller. The controller matches the `my-app` services (matching the run label) on both the local and remote clusters, and consolidates endpoints from the `my-app-ui` TCP port for both of those services. Because the federated service does not specify the `my-app-console` port, the controller does not include these endpoints in the federated service. The endpoints data for the federated service is similar to the following. Note that the name of the remote cluster is included in `targetRef.name`. ```yaml apiVersion: v1 kind: Endpoints metadata: creationTimestamp: 2018-07-03T19:41:38Z annotations: federation.tigera.io/serviceSelector: run == "my-app" name: my-app-federated namespace: default resourceVersion: '701812' selfLink: /api/v1/namespaces/default/endpoints/my-app-federated uid: 1a0427e8-7ef9-11e8-a24c-0259d75c6290 subsets: - addresses: - ip: 192.168.93.12 nodeName: node1.localcluster.tigera.io targetRef: kind: Pod name: my-app-59cf48cdc7-frf2t namespace: default resourceVersion: '701655' uid: 19f5e914-7ef9-11e8-a24c-0259d75c6290 ports: - name: my-app-ui port: 80 protocol: TCP - addresses: - ip: 192.168.0.28 nodeName: node1.remotecluster.tigera.io targetRef: kind: Pod name: remotecluster/my-app-7b6f758bd5-ctgbh namespace: default resourceVersion: '701648' uid: 19e2c841-7ef9-11e8-a24c-0259d75c6290 ports: - name: my-app-ui port: 80 protocol: TCP ``` ## Additional resources[​](#additional-resources) - [Cluster mesh example for AWS](https://docs.tigera.io/calico-cloud/multicluster/aws) - [Federated service controller](https://docs.tigera.io/calico-cloud/reference/component-resources/kube-controllers/configuration) ### Cluster mesh example for clusters in AWS ## Big picture[​](#big-picture) A sample configuration for cluster mesh using AWS clusters. ## Tutorial[​](#tutorial) **Set up** The cluster is installed on real hardware where node and pod IPs are routable, using an edge VPN router to peer with the AWS cluster. ![A diagram showing the key configuration requirements setting up an AWS cluster (using AWS VPN CNI) peering with an on-premise cluster.](https://docs.tigera.io/assets/images/aws-rcc-fdbf3e1f2d8c88bda4097c94b7e52549.svg) **Calico Enterprise configuration** - IP pool resource is configured for the on-premise IP assignment with IPIP is disabled - BGP peering to the VPN router - A Remote Cluster Configuration resource references the AWS cluster - Service discovery of the AWS cluster services uses the Calico Enterprise Federated Services Controller **Notes** - If VPN Router is configured as a route reflector for the on-premise cluster, you would: - Configure the default BGP Configuration resource to disable node-to-node mesh - Configure a global BGP Peer resource to peer with the VPN router - If the IP Pool has `Outgoing NAT` enabled, then you must add an IP Pool covering the AWS cluster VPC with disabled set to `true`. When set to `true` the pool is not used for IP allocations, and SNAT is not performed for traffic to the AWS cluster. **AWS configuration** - A VPC CIDR is chosen that does not overlap with the on-premise IP ranges. - There are 4 subnets within the VPC, split across two AZs (for availability) such that each AZ has a public and private subnet. In this particular example, the split of responsibility is: - The private subnet is used for node and pod IP allocation - The public subnet is used to home a NAT gateway for pod-to-internet traffic. - The VPC is peered to an on-premise network using a VPN. This is configured as a VPN gateway for the AWS side, and a classic VPN for the customer side. BGP is used for route distribution. - Routing table for private subnet has: - "propagate" set to "true" to ensure BGP-learned routes are distributed - Default route to the NAT gateway for public internet traffic - Local VPC traffic - Routing table for public subnet has default route to the internet gateway. - Security group for the worker nodes has: - Rule to allow traffic from the peered networks - Other rules required for settings up VPN peering (refer to the AWS docs for details). To automatically create a Network Load Balancer (NLB) for the AWS deployment, we apply a service with the correct annotation. ```yaml apiVersion: v1 kind: Service metadata: annotations: service.beta.kubernetes.io/aws-load-balancer-type: nlb name: nginx-external spec: externalTrafficPolicy: Local ports: - name: http port: 80 protocol: TCP targetPort: 80 selector: run: nginx type: LoadBalancer ``` --- ## Operations ### Operations Post-installation tasks for managing Calico Cloud. ##### [Manage clusters](https://docs.tigera.io/calico-cloud/operations/cluster-management) [Manage cluster storage, add new clusters, and configure alerts.](https://docs.tigera.io/calico-cloud/operations/cluster-management) ##### [Uninstall Calico Cloud from a cluster](https://docs.tigera.io/calico-cloud/operations/disconnect) [Steps to use migration script to uninstall Calico Cloud from a cluster.](https://docs.tigera.io/calico-cloud/operations/disconnect) ##### [Calico Cloud Pro usage and billing](https://docs.tigera.io/calico-cloud/operations/usage-metrics) [Where to find Calico Cloud usage metrics.](https://docs.tigera.io/calico-cloud/operations/usage-metrics) ##### [Component logs](https://docs.tigera.io/calico-cloud/operations/component-logs) [Where to find component logs.](https://docs.tigera.io/calico-cloud/operations/component-logs) ## Secure component communications[​](#secure-component-communications) ##### [Secure Calico Cloud Prometheus endpoints](https://docs.tigera.io/calico-cloud/operations/comms/secure-metrics) [Limit access to Calico Cloud metric endpoints using network policy.](https://docs.tigera.io/calico-cloud/operations/comms/secure-metrics) ##### [Secure BGP sessions](https://docs.tigera.io/calico-cloud/operations/comms/secure-bgp) [Configure BGP passwords to prevent attackers from injecting false routing information.](https://docs.tigera.io/calico-cloud/operations/comms/secure-bgp) ## Monitoring[​](#monitoring) ##### [Prometheus support](https://docs.tigera.io/calico-cloud/operations/monitor/prometheus/support) [Prometheus support in Calico Cloud.](https://docs.tigera.io/calico-cloud/operations/monitor/prometheus/support) ##### [Recommended Prometheus metrics](https://docs.tigera.io/calico-cloud/operations/monitor/metrics/recommended-metrics) [Recommended Prometheus metrics for monitoring Calico Enterprise components.](https://docs.tigera.io/calico-cloud/operations/monitor/metrics/recommended-metrics) ##### [Bring your own Prometheus](https://docs.tigera.io/calico-cloud/operations/monitor/prometheus/byo-prometheus) [Steps to get Calico Cloud metrics using your own Prometheus.](https://docs.tigera.io/calico-cloud/operations/monitor/prometheus/byo-prometheus) ##### [Configure Prometheus](https://docs.tigera.io/calico-cloud/operations/monitor/prometheus/configure-prometheus) [Configure rules for alerts and denied packets, for persistent storage.](https://docs.tigera.io/calico-cloud/operations/monitor/prometheus/configure-prometheus) ##### [Configure Alertmanager](https://docs.tigera.io/calico-cloud/operations/monitor/prometheus/alertmanager) [Configure Alertmanager, a Prometheus feature that routes alerts.](https://docs.tigera.io/calico-cloud/operations/monitor/prometheus/alertmanager) ##### [BGP metrics](https://docs.tigera.io/calico-cloud/operations/monitor/metrics/bgp-metrics) [Monitor BGP peering and route exchange in your cluster and get alerts by defining rules and thresholds.](https://docs.tigera.io/calico-cloud/operations/monitor/metrics/bgp-metrics) ##### [Policy metrics](https://docs.tigera.io/calico-cloud/operations/monitor/metrics/policy-metrics) [Monitor the effects of policy in your cluster and received alerts by defining rules and thresholds.](https://docs.tigera.io/calico-cloud/operations/monitor/metrics/policy-metrics) ##### [Fluentd metrics](https://docs.tigera.io/calico-cloud/operations/monitor/metrics/elasticsearch-and-fluentd-metrics) [Monitor Fluentd metrics and get alerts on log storage or collection issues.](https://docs.tigera.io/calico-cloud/operations/monitor/metrics/elasticsearch-and-fluentd-metrics) ## eBPF[​](#ebpf) ##### [eBPF use cases](https://docs.tigera.io/calico-cloud/operations/ebpf/use-cases-ebpf) [Learn when to use eBPF, and when not to.](https://docs.tigera.io/calico-cloud/operations/ebpf/use-cases-ebpf) ##### [Enable eBPF on an existing cluster](https://docs.tigera.io/calico-cloud/operations/ebpf/enabling-ebpf) [Steps to enable the eBPF data plane on an existing cluster.](https://docs.tigera.io/calico-cloud/operations/ebpf/enabling-ebpf) ##### [Troubleshoot eBPF mode](https://docs.tigera.io/calico-cloud/operations/ebpf/troubleshoot-ebpf) [How to troubleshoot when running in eBPF mode.](https://docs.tigera.io/calico-cloud/operations/ebpf/troubleshoot-ebpf) ### Manage clusters ## Connect new clusters[​](#connect-new-clusters) To connect new clusters to Calico Cloud, select Managed Clusters in the left nav, and click **Connect Cluster**. ## Remove a cluster from the managed clusters list[​](#remove-a-cluster-from-the-managed-clusters-list) ### Prerequisites[​](#prerequisites) - You are signed in to the web console and you have [Owner or Admin privileges](https://docs.tigera.io/calico-cloud/users/user-management). - The cluster you want to remove is [disconnected from Calico Cloud](https://docs.tigera.io/calico-cloud/operations/disconnect). ### Remove a cluster[​](#remove-a-cluster) For the cluster you want to remove, select **Actions** > **Delete**, and then confirm your choice by selecting **Delete cluster**. > **SECONDARY:** Deleting a managed cluster does not remove Calico Cloud logs. > > If a new managed cluster is connected with the same name as a previously deleted managed cluster, the logs associated with the deleted managed cluster will be displayed as if they belong to the new managed cluster. ### Recover a deleted managed cluster[​](#recover-a-deleted-managed-cluster) If you inadvertently delete a managed cluster, you may restore it by connecting your cluster using the same name as the previously deleted cluster. For help with connecting a managed cluster see [Connect a cluster to Calico Cloud](https://docs.tigera.io/calico-cloud/get-started/install-cluster). ## Manage and monitor multiple clusters in the web console[​](#manage-and-monitor-multiple-clusters-in-the-web-console) ### Access your managed clusters[​](#access-your-managed-clusters) Each time you install Calico Cloud on a cluster, the cluster is added to the drop-down menu in the web console. ![cluster-menu](https://docs.tigera.io/assets/images/cluster-menu-1215464378fce77214508e9bef8d7e16.png) Use this menu to move between your clusters. You can connect as many managed clusters as your subscriptions allows. ### Managed cluster views[​](#managed-cluster-views) When you select a different cluster from the cluster menu, the entire web console view changes to reflect the selected cluster. For example, each managed cluster has unique indexes for Elasticsearch clusters, a separate instance of Kibana, and the dashboard reflects only the selected managed cluster. ### Manage policies in multiple clusters[​](#manage-policies-in-multiple-clusters) To implement, modify, or view network policies in a given cluster: 1. Toggle to a managed cluster using the managed cluster drop-down menu. 2. Define, modify or view network policies in that cluster. 3. You can monitor the impact of a policy using the policy visualization tool (Flow Viz). To apply policies in one managed cluster to a different cluster, manually apply the manifest/.yaml using `kubectl`. To download manifests for a policy, click **Export** for the selected policy. ![export-policy](https://docs.tigera.io/assets/images/export-policy-22004dd4cde380f4be0ff329f2c4cec6.png) ### Uninstall Calico Cloud from a cluster Whether you've finished with your Calico Cloud Trial or decided to disconnect your cluster from Calico Cloud, we know you want your cluster to remain functional. We highly recommend running a simple script to migrate your cluster to open-source Project Calico. ## About the migration script[​](#about-the-migration-script) The script migrates all applicable Calico Cloud components to open-source Project Calico; this includes removal and cleanup of all Calico Cloud components that have no equivalents in Project Calico. Because Project Calico does not have the tier resource, the script will exit if any policies exist in any tier except for the `default` or `allow-tigera` tiers. To remove policies from tiers, you have these options: - Manually move policies out of tiers prior to running the script - Let the script remove *ALL* Calico policies by specifying the `--remove-all-calico-policy` flag > **SECONDARY:** To successfully downgrade to an open-source Calico configuration, policies must allow necessary traffic to and from Calico Cloud and open-source Calico namespaces. If you keep policies in the default tier (especially GlobalNetworkPolicies), and you have default deny policies, update or add policies accordingly to allow this necessary traffic. > **WARNING:** If your cluster began with Calico installed and managed by AKS with AddonManager, this uninstall process will not be successful. You will need to reach out to your support contact to create a plan to uninstall Calico Cloud. ### Before you begin[​](#before-you-begin) - You have `kubectl` administrator access to the cluster you want to migrate to Calico Open Source. - You are accessing the cluster from a Linux-based machine. ### Run the migration script[​](#run-the-migration-script) 1. Download the script `curl -O https://installer.calicocloud.io/manifests/v3.22.0-2.0-7/downgrade.sh`. 2. Make the script executable `chmod +x downgrade.sh`. 3. Run the script and read the help to determine if you need to specify any flags `./downgrade.sh --help`. 4. Run the script with any needed flags, for example: `./downgrade.sh --remove-prometheus`. ## Next steps[​](#next-steps) Continue using your cluster with open-source [Project Calico](https://docs.tigera.io/calico/latest/about). ### Calico Cloud Pro usage and billing To determine your monthly usage, we calculate the vCPU usage time in hours for your connected clusters. ## Metering unit and standard pricing[​](#metering-unit-and-standard-pricing) Usage is calculated monthly as a number of *vCPU hours*. This includes the total vCPU hours for all your connected clusters. The standard pricing is US$0.025 per vCPU hour. There is no additional cost for log data ingestion. ## View your usage metrics[​](#view-your-usage-metrics) To view usage metrics, Calico Cloud administrators can go to the web console, click the user icon , and then click **Usage Metrics**. ![A screenshot of the Usage Metrics page in the Calico Cloud web console.](https://docs.tigera.io/img/calico-cloud/usage-metrics.png) ## Subscription models[​](#subscription-models) | Subscription Model | Pricing | Usage | Additional Usage | | ------------------ | ------------------------- | --------------------------------- | -------------------------------------------------------------------------------- | | **Monthly** | US$0.025 per vCPU hour | Pay for actual usage each month | Billed at the same rate | | **Annual** | Prepaid set of vCPU hours | Usage deducted from prepaid hours | Additional hours can be purchased; excess usage billed at US$0.025 per vCPU hour | Example burndown table for an annual subscription The following table shows a the month-to-month usage for a cluster with an annual allocation of 12,000 vCPU hours. | Month | Starting Balance (vCPU hours) | Hours Consumed | Ending Balance (vCPU hours) | | --------- | ----------------------------- | -------------- | --------------------------- | | January | 12,000 | 1,000 | 11,000 | | February | 11,000 | 900 | 10,100 | | March | 10,100 | 1,200 | 8,900 | | April | 8,900 | 950 | 7,950 | | May | 7,950 | 1,050 | 6,900 | | June | 6,900 | 1,000 | 5,900 | | July | 5,900 | 1,200 | 4,700 | | August | 4,700 | 1,100 | 3,600 | | September | 3,600 | 900 | 2,700 | | October | 2,700 | 1,300 | 1,400 | | November | 1,400 | 1,200 | 200 | | December | 200 | 250 | -50 (overage applies) | > **SECONDARY:** If you're approaching your usage limit, you can purchase additional vCPU hours to avoid overage charges. Otherwise, overage charges apply at the standard rate of US$0.025 per vCPU hour. ### Secure Calico component communications ## [📄️ Secure Calico Cloud Prometheus endpoints](https://docs.tigera.io/calico-cloud/operations/comms/secure-metrics) [Limit access to Calico Cloud metric endpoints using network policy.](https://docs.tigera.io/calico-cloud/operations/comms/secure-metrics) ## [📄️ Secure BGP sessions](https://docs.tigera.io/calico-cloud/operations/comms/secure-bgp) [Configure BGP passwords to prevent attackers from injecting false routing information.](https://docs.tigera.io/calico-cloud/operations/comms/secure-bgp) ### Secure Calico Cloud Prometheus endpoints ## About securing access to Calico Cloud's metrics endpoints[​](#about-securing-access-to-calico-clouds-metrics-endpoints) When using Calico Cloud with Prometheus metrics enabled, we recommend using network policy to limit access to Calico Cloud's metrics endpoints. ## Prerequisites[​](#prerequisites) - Calico Cloud is installed with Prometheus metrics reporting enabled. ## Choosing an approach[​](#choosing-an-approach) This guide provides two example workflows for creating network policies to limit access to Calico Cloud's Prometheus metrics. Choosing an approach depends on your requirements. - [Using a deny-list approach](#using-a-deny-list-approach) This approach allows all traffic to your hosts by default, but lets you limit access to specific ports using Calico Cloud policy. This approach allows you to restrict access to specific ports, while leaving other host traffic unaffected. - [Using an allow-list approach](#using-an-allow-list-approach) This approach denies traffic to and from your hosts by default, and requires that all desired communication be explicitly allowed by a network policy. This approach is more secure because only explicitly-allowed traffic will get through, but it requires you to know all the ports that should be open on the host. ## Using a deny-list approach[​](#using-a-deny-list-approach) ### Overview[​](#overview) The basic process is as follows: 1. Create a default network policy that allows traffic to and from your hosts. 2. Create host endpoints for each node that you'd like to secure. 3. Create a network policy that denies unwanted traffic to the Calico Cloud metrics endpoints. 4. Apply labels to allow access to the Prometheus metrics. ### Example for cnx-node[​](#example-for-cnx-node) This example shows how to limit access to the cnx-node Prometheus metrics endpoints. 1. Create a default network policy to allow host traffic First, create a default-allow policy. Do this first to avoid a drop in connectivity when adding the host endpoints later, since host endpoints with no policy default to deny. To do this, create a file named `default-host-policy.yaml` with the following contents. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: default-host spec: # Select all Calico Cloud nodes. selector: running-calico == "true" order: 5000 ingress: - action: Allow egress: - action: Allow ``` Then, use `kubectl` to apply this policy. ```bash kubectl apply -f default-host-policy.yaml ``` 2. List the nodes on which Calico Cloud is running with the following command. ```bash calicoctl get nodes ``` In this case, we have two nodes in the cluster. ```text NAME kubeadm-master kubeadm-node-0 ``` 3. Create host endpoints for each Calico Cloud node. Create a file named `host-endpoints.yaml` containing a host endpoint for each node listed above. In this example, the contents would look like this. ```yaml apiVersion: projectcalico.org/v3 kind: HostEndpoint metadata: name: kubeadm-master.eth0 labels: running-calico: 'true' spec: node: kubeadm-master interfaceName: eth0 expectedIPs: - 10.100.0.15 --- apiVersion: projectcalico.org/v3 kind: HostEndpoint metadata: name: kubeadm-node-0.eth0 labels: running-calico: 'true' spec: node: kubeadm-node-0 interfaceName: eth0 expectedIPs: - 10.100.0.16 ``` In this file, replace `eth0` with the desired interface name on each node, and populate the `expectedIPs` section with the IP addresses on that interface. Note the use of a label to indicate that this host endpoint is running Calico Cloud. The label matches the selector of the network policy created in step 1. Then, use `kubectl` to apply the host endpoints with the following command. ```bash kubectl apply -f host-endpoints.yaml ``` 4. Create a network policy that restricts access to the cnx-node Prometheus metrics port. Now let's create a network policy that limits access to the Prometheus metrics port such that only endpoints with the label `calico-prometheus-access: true` can access the metrics. To do this, create a file named `calico-prometheus-policy.yaml` with the following contents. ```yaml # Allow traffic to Prometheus only from sources that are # labeled as such, but don't impact any other traffic. apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: restrict-calico-node-prometheus spec: # Select all Calico Cloud nodes. selector: running-calico == "true" order: 500 types: - Ingress ingress: # Deny anything that tries to access the Prometheus port # but that doesn't match the necessary selector. - action: Deny protocol: TCP source: notSelector: calico-prometheus-access == "true" destination: ports: - 9091 ``` This policy selects all endpoints that have the label `running-calico: true`, and enforces a single ingress deny rule. The ingress rule denies traffic to port 9091 unless the source of traffic has the label `calico-prometheus-access: true`, meaning all Calico Cloud workload endpoints, host endpoints, and global network sets that do not have the label, as well as any other network endpoints unknown to Calico Cloud. Then, use `kubectl` to apply this policy. ```bash kubectl apply -f calico-prometheus-policy.yaml ``` 5. Apply labels to any endpoints that should have access to the metrics. At this point, only endpoints that have the label `calico-prometheus-access: true` can reach Calico Cloud's Prometheus metrics endpoints on each node. To grant access, simply add this label to the desired endpoints. For example, to allow access to a Kubernetes pod you can run the following command. ```bash kubectl label pod my-prometheus-pod calico-prometheus-access=true ``` If you would like to grant access to a specific IP network, you can create a [global network set](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkset) using `kubectl`. For example, you might want to grant access to your management subnets. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkSet metadata: name: calico-prometheus-set labels: calico-prometheus-access: 'true' spec: nets: - 172.15.0.0/24 - 172.101.0.0/24 ``` ### Additional steps for Typha deployments[​](#additional-steps-for-typha-deployments) If your Calico Cloud installation uses the Kubernetes API datastore and has greater than 50 nodes, it is likely that you have installed Typha. This section shows how to use an additional network policy to secure the Typha Prometheus endpoints. After following the steps above, create a file named `typha-prometheus-policy.yaml` with the following contents. ```yaml # Allow traffic to Prometheus only from sources that are # labeled as such, but don't impact any other traffic. apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: restrict-calico-node-prometheus spec: # Select all Calico Cloud nodes. selector: running-calico == "true" order: 500 types: - Ingress ingress: # Deny anything that tries to access the Prometheus port # but that doesn't match the necessary selector. - action: Deny protocol: TCP source: notSelector: calico-prometheus-access == "true" destination: ports: - 9093 ``` This policy selects all endpoints that have the label `running-calico: true`, and enforces a single ingress deny rule. The ingress rule denies traffic to port 9093 unless the source of traffic has the label `calico-prometheus-access: true`, meaning all Calico Cloud workload endpoints, host endpoints, and global network sets that do not have the label, as well as any other network endpoints unknown to Calico Cloud. Then, use `kubectl` to apply this policy. ```bash kubectl apply -f typha-prometheus-policy.yaml ``` ### Example for kube-controllers[​](#example-for-kube-controllers) If your Calico Cloud installation exposes metrics from kube-controllers, you can limit access to those metrics with the following network policy. Create a file named `kube-controllers-prometheus-policy.yaml` with the following contents. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: restrict-kube-controllers-prometheus namespace: calico-system spec: # Select kube-controllers. selector: k8s-app == "calico-kube-controllers" order: 500 types: - Ingress ingress: # Deny anything that tries to access the Prometheus port # but that doesn't match the necessary selector. - action: Deny protocol: TCP source: notSelector: calico-prometheus-access == "true" destination: ports: - 9094 ``` > **SECONDARY:** The above policy is installed in the calico-system namespace. If your cluster has Calico Cloud installed in the kube-system namespace, you will need to create the policy in that namespace instead. Then, use `calicoctl` to apply this policy. ```bash kubectl apply -f kube-controllers-prometheus-policy.yaml ``` ## Using an allow-list approach[​](#using-an-allow-list-approach) ### Overview[​](#overview-1) The basic process is as follows: 1. Create host endpoints for each node that you'd like to secure. 2. Create a network policy that allows desired traffic to the Calico Cloud metrics endpoints. 3. Apply labels to allow access to the Prometheus metrics. ### Example for cnx-node[​](#example-for-cnx-node-1) 1. List the nodes on which Calico Cloud is running with the following command. ```bash calicoctl get nodes ``` In this case, we have two nodes in the cluster. ```text NAME kubeadm-master kubeadm-node-0 ``` 2. Create host endpoints for each Calico Cloud node. Create a file named `host-endpoints.yaml` containing a host endpoint for each node listed above. In this example, the contents would look like this. ```yaml apiVersion: projectcalico.org/v3 kind: HostEndpoint metadata: name: kubeadm-master.eth0 labels: running-calico: 'true' spec: node: kubeadm-master interfaceName: eth0 expectedIPs: - 10.100.0.15 --- apiVersion: projectcalico.org/v3 kind: HostEndpoint metadata: name: kubeadm-node-0.eth0 labels: running-calico: 'true' spec: node: kubeadm-node-0 interfaceName: eth0 expectedIPs: - 10.100.0.16 ``` In this file, replace `eth0` with the desired interface name on each node, and populate the `expectedIPs` section with the IP addresses on that interface. Note the use of a label to indicate that this host endpoint is running Calico Cloud. The label matches the selector of the network policy created in step 1. Then, use `kubectl` to apply the host endpoints with the following command. This will prevent all traffic to and from the host endpoints. ```bash kubectl apply -f host-endpoints.yaml ``` > **SECONDARY:** Calico Cloud allows some traffic as a failsafe even after applying this policy. This can be adjusted using the `failsafeInboundHostPorts` and `failsafeOutboundHostPorts` options on the [FelixConfiguration resource](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig). 3. Create a network policy that allows access to the cnx-node Prometheus metrics port. Now let's create a network policy that allows access to the Prometheus metrics port such that only endpoints with the label `calico-prometheus-access: true` can access the metrics. To do this, create a file named `calico-prometheus-policy.yaml` with the following contents. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: restrict-calico-node-prometheus spec: # Select all Calico Cloud nodes. selector: running-calico == "true" order: 500 types: - Ingress ingress: # Allow traffic from selected sources to the Prometheus port. - action: Allow protocol: TCP source: selector: calico-prometheus-access == "true" destination: ports: - 9091 ``` This policy selects all endpoints that have the label `running-calico: true`, and enforces a single ingress allow rule. The ingress rule allows traffic to port 9091 from any source with the label `calico-prometheus-access: true`, meaning all Calico Cloud workload endpoints, host endpoints, and global network sets that have the label will be allowed access. Then, use `kubectl` to apply this policy. ```bash kubectl apply -f calico-prometheus-policy.yaml ``` 4. Apply labels to any endpoints that should have access to the metrics. At this point, only endpoints that have the label `calico-prometheus-access: true` can reach Calico Cloud's Prometheus metrics endpoints on each node. To grant access, simply add this label to the desired endpoints. For example, to allow access to a Kubernetes pod you can run the following command. ```bash kubectl label pod my-prometheus-pod calico-prometheus-access=true ``` If you would like to grant access to a specific IP address in your network, you can create a [global network set](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkset) using `kubectl`. For example, creating the following network set would grant access to a host with IP 172.15.0.101. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkSet metadata: name: calico-prometheus-set labels: calico-prometheus-access: 'true' spec: nets: - 172.15.0.101/32 ``` ### Additional steps for Typha deployments[​](#additional-steps-for-typha-deployments-1) If your Calico Cloud installation uses the Kubernetes API datastore and has greater than 50 nodes, it is likely that you have installed Typha. This section shows how to use an additional network policy to secure the Typha Prometheus endpoints. After following the steps above, create a file named `typha-prometheus-policy.yaml` with the following contents. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: restrict-typha-prometheus spec: # Select all Calico Cloud nodes. selector: running-calico == "true" order: 500 types: - Ingress ingress: - action: Allow protocol: TCP source: selector: calico-prometheus-access == "true" destination: ports: - 9093 ``` This policy selects all endpoints that have the label `running-calico: true`, and enforces a single ingress allow rule. The ingress rule allows traffic to port 9093 from any source with the label `calico-prometheus-access: true`, meaning all Calico Cloud workload endpoints, host endpoints, and global network sets that have the label will be allowed access. Then, use `kubectl` to apply this policy. ```bash kubectl apply -f typha-prometheus-policy.yaml ``` ### Example for kube-controllers[​](#example-for-kube-controllers-1) If your Calico Cloud installation exposes metrics from kube-controllers, you can limit access to those metrics with the following network policy. Create a file named `kube-controllers-prometheus-policy.yaml` with the following contents. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: restrict-kube-controllers-prometheus namespace: calico-system spec: selector: k8s-app == "calico-kube-controllers" order: 500 types: - Ingress ingress: - action: Allow protocol: TCP source: selector: calico-prometheus-access == "true" destination: ports: - 9094 ``` Then, use `kubectl` to apply this policy. ```bash kubectl apply -f kube-controllers-prometheus-policy.yaml ``` ### Secure BGP sessions ## Big picture[​](#big-picture) Use BGP passwords to prevent attackers from injecting false routing information. ## Value[​](#value) Setting a password on a BGP peering between BGP speakers means that a peering will only work when both ends of the peering have the same password. This provides a layer of defense against an attacker impersonating an external BGP peer or a workload in the cluster, for example to inject malicious routing information into the cluster. ## Concepts[​](#concepts) ### Password protection on BGP sessions[​](#password-protection-on-bgp-sessions) Password protection is a [standardized](https://tools.ietf.org/html/rfc5925) optional feature of BGP sessions. The effect is that the two peers at either end of a BGP session can only communicate, and exchange routing information, if they are both configured with the same password. Please note that password use does not cause the data exchange to be *encrypted*. It remains relatively easy to *eavesdrop* on the data exchange, but not to *inject* false information. ### Using Kubernetes secrets to store passwords[​](#using-kubernetes-secrets-to-store-passwords) In Kubernetes, the Secret resource is designed for holding sensitive information, including passwords. Therefore, for this Calico Cloud feature, we use Secrets to store BGP passwords. ## How to[​](#how-to) To use a password on a BGP peering: 1. Create (or update) a Kubernetes secret in the namespace where calico-node is running, so that it has a key whose value is the desired password. Note the secret name and the key name. > **SECONDARY:** BGP passwords must be 80 characters or fewer. If a password longer than that is configured, the BGP sessions with that password will fail to be established. 2. Ensure that calico-node has RBAC permissions to access that secret. 3. Specify the secret and key name on the relevant BGPPeer resource. ### Create or update Kubernetes secret[​](#create-or-update-kubernetes-secret) For example: ```text kubectl create -f - < **SECONDARY:** If calico-node in your cluster is running in a namespace other than calico-system, you should create the secret in that namespace instead of in calico-system. To use this password below in a BGPPeer resource, you need to note the secret name `bgp-secrets` and key name `rr-password`. ### Ensure RBAC permissions[​](#ensure-rbac-permissions) The calico-node pod must have permission to access that secret. To allow calico-node to access that secret, you would configure: ```text kubectl create -f - < **Tab: Specific or external peerings** When [configuring a BGP peer](https://docs.tigera.io/calico-cloud/networking/configuring/bgp), include the secret and key name in the specification of the BGPPeer resource, like this: ```yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: bgppeer-global-3040 spec: peerIP: 192.20.30.40 asNumber: 64567 password: secretKeyRef: name: bgp-secrets key: rr-password ``` **Tab: Node to node mesh** Include the secret in the default [BGP configuration](https://docs.tigera.io/calico-cloud/reference/resources/bgpconfig) similar to the following: ```yaml kind: BGPConfiguration apiVersion: projectcalico.org/v3 metadata: name: default spec: logSeverityScreen: Info nodeToNodeMeshEnabled: true nodeMeshPassword: secretKeyRef: name: bgp-secrets key: rr-password ``` > **SECONDARY:** Node to node mesh must be enabled to set node to node mesh BGP password. ## Additional resources[​](#additional-resources) For more detail about the BGPPeer resource, see [BGPPeer](https://docs.tigera.io/calico-cloud/reference/resources/bgppeer). For more on configuring BGP peers, see [configuring BGP peers](https://docs.tigera.io/calico-cloud/networking/configuring/bgp) . ### Monitoring ## [🗃️ Prometheus](https://docs.tigera.io/calico-cloud/operations/monitor/prometheus/) [5 items](https://docs.tigera.io/calico-cloud/operations/monitor/prometheus/) ## [🗃️ Metrics](https://docs.tigera.io/calico-cloud/operations/monitor/metrics/) [3 items](https://docs.tigera.io/calico-cloud/operations/monitor/metrics/) ### Prometheus ## [📄️ Prometheus support](https://docs.tigera.io/calico-cloud/operations/monitor/prometheus/support) [Prometheus support in Calico Cloud.](https://docs.tigera.io/calico-cloud/operations/monitor/prometheus/support) ## [📄️ Recommended Prometheus metrics](https://docs.tigera.io/calico-cloud/operations/monitor/metrics/recommended-metrics) [Recommended Prometheus metrics for monitoring Calico Enterprise components.](https://docs.tigera.io/calico-cloud/operations/monitor/metrics/recommended-metrics) ## [📄️ Bring your own Prometheus](https://docs.tigera.io/calico-cloud/operations/monitor/prometheus/byo-prometheus) [Steps to get Calico Cloud metrics using your own Prometheus.](https://docs.tigera.io/calico-cloud/operations/monitor/prometheus/byo-prometheus) ## [📄️ Configure Prometheus](https://docs.tigera.io/calico-cloud/operations/monitor/prometheus/configure-prometheus) [Configure rules for alerts and denied packets, for persistent storage.](https://docs.tigera.io/calico-cloud/operations/monitor/prometheus/configure-prometheus) ## [📄️ Configure Alertmanager](https://docs.tigera.io/calico-cloud/operations/monitor/prometheus/alertmanager) [Configure Alertmanager, a Prometheus feature that routes alerts.](https://docs.tigera.io/calico-cloud/operations/monitor/prometheus/alertmanager) ### Prometheus support ## Big picture[​](#big-picture) Calico Cloud uses the open-source [Prometheus monitoring and alerting toolkit](https://prometheus.io/docs/introduction/overview/). With these tools, you can view time-series metrics from Calico Cloud components in the Prometheus and Grafana interfaces, or scrape the metrics for a BYO Prometheus deployment. ## Install options[​](#install-options) - Use Prometheus operator managed by Tigera Operator You install the Calico Cloud Prometheus operator and CRDs during Calico Cloud installation. Calico Cloud metrics and alerts are available in the web console. You configure alerts through Prometheus AlertManager. If you want to specify your own Prometheus operator during installation for management by the Tigera Operator, the require operator version must be **v0.40.0 or higher**. Because Calico Cloud creates AlertManager and Prometheus CRs in the `tigera-prometheus` namespace, all you need to do is verify that your Prometheus operator is configured to manage Prometheus and AlertManager instances in the `tigera-prometheus` namespace. - [Bring your own Prometheus](https://docs.tigera.io/calico-cloud/operations/monitor/prometheus/byo-prometheus) ### Recommended Prometheus metrics ## Big picture[​](#big-picture) Monitor the Calico Cloud Typha, Felix, and policy component metrics to ensure optimal cluster operation. ## Concepts[​](#concepts) Calico Cloud Typha, Felix, and policy components are the most critical to monitor because they are responsible for ensuring networking and security functions are up-to-date and working as expected. ### Typha[​](#typha) Typha is a caching datastore proxy that sits between calico-nodes and Kubernetes API Server. Its primary function is to allow for increased cluster scale by reducing the load on Kubernetes API Server. Without Typha, large clusters (200+ nodes) would need a considerable amount of memory to correspond to the continuous watches and requests from calico-nodes running in the cluster. Typha maintains a single datastore connection on behalf of all of its clients (processes running in the calico-node pods, with Felix being Typhas’ main client). Typha watches for node, pod, network policy, bgp configuration, and other events on the Kubernetes API Server, caches and deduplicates this data, and fans out these events to its clients. ### Felix[​](#felix) Felix is a component of calico-node and is responsible for Calico Cloud network policy. Felix must be continuously in sync with the datastore to ensure the correct set of policies are applied to the node it is running on. ![Typha-felix](https://docs.tigera.io/assets/images/typha-felix-d57d0ce5298216dc54e9d42241235242.png) ### About metrics[​](#about-metrics) Each Calico Cloud component that you want to connect to Prometheus for endpoint metrics has its own configuration (bgp, license, policy, felix, and typha). Note that Felix is a separate application with metric endpoints, its own core metrics to monitor itself, and a separate port for a second policy metric endpoint. ## Metrics[​](#metrics) This section provides metrics recommendations for maintaining optimal cluster operations. Note the following: - Threshold values for each metric depend on the cluster size and churn rate. - Threshold recommendations are provided where possible, but because each cluster is different, and metrics can depend on cluster churn rate and scale. We recommend that you baseline the cluster to establish numbers that represent normal figures for your cluster. - Metrics that start increasing rapidly from the baseline set need attention. **Typha** - [Typha general metrics](#typha-general-metrics) - [Typha cluster mesh metrics](#typha-cluster-mesh-metrics) - [Typha client metrics](#typha-client-metrics) - [Typha cache internals](#typha-cache-internals) - [Typha snapshot details](#typha-snapshot-details) **Felix** - [Policy metrics](#policy-metrics) - [Felix cluster state metrics](#felix-cluster-state-metrics) - [Felix error metrics](#felix-error-metrics) - [Felix time-based metrics](#felix-time-based-metrics) ## Typha general metrics[​](#typha-general-metrics) ### Datastore cache size[​](#datastore-cache-size) | Datastore cache size | | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Metric | **Note**: Syncer (type) is Typha's internal name for a client (type). **Individual syncer values**: `(typha_cache_size{syncer="bgp"})` `(typha_cache_size{syncer="dpi"})` `(typha_cache_size{syncer="felix"})` `(typha_cache_size{syncer="node-status"})` `(typha_cache_size{syncer="tunnel-ip-allocation"})` **Sum of all syncers**: The sum of all cache sizes (each syncer type has a cache). `sum by (instance)` `(typha_cache_size)` **Largest syncer**: `max by (instance)` `(typha_cache_size)` | | Example value | Example of: `max by (instance)` `(typha_cache_size{syncer="felix"})` `{instance="10.0.1.20:9093"} 661` `{instance="10.0.1.31:9093"} 661` | | Explanation | The total number of key/value pairs in Typha's in-memory cache.This metric represents the scale of the Calico Cloud datastore as it tracks how many WEPs (pods and services), HEPs (hostendpoints), networksets, globalnetworksets, Calico Cloud Network Policies etc that Typha is aware of across the entire Calico Federation.You can use this metric to monitor individual syncers to Typha (like Felix, BGP etc), or to get a sum of all syncers. We recommend that you monitor the largest syncer but it is completely up to you. This is a good metric to understand how much data is in Typha. **Note**: If all Typhas are in sync then they should have the same value for this metric. | | Threshold value recommendation | The value of this metric will depend on the scale of the Calico Federation and will always increase as WEPs, Calico Cloud network policies and clusters are added. Achieve a baseline first, then monitor for any unexpected increases from the baseline. | | Threshold breach symptoms | Unexpected increases may indicate memory leaks and performance issues with Typha. | | Threshold breach recommendations | Check CPU usage on Typha pods and Kubernetes nodes. Increase resources if needed, rollout and restart Typha(s) if needed. | | Priority level | Optional. | ### CPU usage[​](#cpu-usage) | CPU usage | | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Metric | `rate(process_cpu_seconds_total{30s}) * 100` | | Example value | `{endpoint="metrics-port", instance="10.0.1.20:9093", job="typha-metrics-svc", namespace="calico-system", pod="calico-typha-6c6cc9fcf7-csbdl", service="typha-metrics-svc"} 0.27999999999999403` | | Explanation | CPU in use by Typha represented as a percentage of a core. | | Threshold value recommendation | A spike at startup is normal. It is recommended to achieve a baseline first, then monitor for any unexpected increases from this baseline. A rule of thumb is to investigate maintained CPU usage above 90%. | | Threshold breach symptoms | Unexpected maintained CPU usage could cause Typha to fall behind in updating its clients (for example, Felix) and could cause delays to policy updates. | | Threshold breach recommendations | Check CPU usage on Kubernetes nodes. If needed, increase resources, and rollout restart Typha(s). | | Priority level | Recommended. | ### Memory usage[​](#memory-usage) | Memory usage | | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Metric | `process_resident_memory_bytes` | | Example value | `process_resident_memory_bytes{endpoint="metrics-port", instance="10.0.1.20:9093", job="typha-metrics-svc", namespace="calico-system", pod="calico-typha-6c6cc9fcf7-csbdl", service="typha-metrics-svc"} 80515072` | | Explanation | Amount of memory used by Typha. | | Threshold value recommendation | It is recommended to achieve a baseline first, then monitor for any unexpected increases from this baseline. A rule of thumb is to investigate if maintained memory usage is above 90% of what is available from the underlying node. The metric can also be used for memory leaks. In this case, the metric would show Typhas' memory consumption rising over time, even though the cluster is in a stable state. | | Threshold breach symptoms | Unexpected maintained memory usage could cause Typha to fall behind in updating its clients (for example, Felix) and could cause delays to policy updates. | | Threshold breach recommendations | Check memory usage on Kubernetes nodes. Increase resources if needed, and rollout restart Typha(s) if needed. | | Priority level | Recommended. | ## Typha cluster mesh metrics[​](#typha-cluster-mesh-metrics) The following metrics are applicable only if you have implemented [Cluster mesh](https://docs.tigera.io/calico-cloud/multicluster/overview). Note that this metric requires a count syntax because you will have a copy of the metric per RemoteClusterConfiguration. As shown in the table, the value `2 = In Sync` reflects good connections. ```text remote_cluster_connection_status\{cluster="foo"\} = 2 remote_cluster_connection_status\{cluster="bar"\} = 2 remote_cluster_connection_status\{cluster="baz"\} = 1 ``` ### Remote cluster connections (in-sync)[​](#remote-cluster-connections-in-sync) | Remote cluster connections (in-sync) | | | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `count by (instance) (remote_cluster_connection_status == 2)` | | Explanation | This represents the number of remote cluster connections that are connected and in sync. Each remote cluster will report a *connection\_status* value from the following list: - 0 = Not Connected - 1 = Connecting - **2 = In Sync** - 3 = Resync in Process - 4 = Config Change Restart Required We suggest the **count** syntax because there will be one copy of *remote\_cluster\_connection\_status* per cluster: - `remote_cluster_connection_status[cluster="foo"] = 2` `remote_cluster_connection_status[cluster="bar"] = 2` `remote_cluster_connection_status[cluster="baz"] = `' Counting the number of metrics with value **2** returns the number of **In Sync** clusters. | | Threshold value recommendation | When remote cluster connections are initializing, *connection\_status* values will fluctuate. After the connection is established, this value should be equal to the number of remote clusters in the environment (if everything is in sync). | | Threshold breach symptoms | N/A For out-of-sync symptoms, see the out-of-sync metric. | | Threshold breach recommendations | N/A For out-of-sync recommendations, see the out-of-sync metric. | | Priority level | Recommended. | ### Remote cluster connections (out-of-sync)[​](#remote-cluster-connections-out-of-sync) The following metrics are applicable only if you have implemented [Cluster mesh](https://docs.tigera.io/calico-cloud/multicluster/overview). | Remote cluster connections (out-of-sync) | | | ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `count by (instance) (remote_cluster_connection_status != 2)` | | Explanation | Number of remote cluster connections that are **not** in sync (i.e. resyncing or failing to connect). Each remote cluster will report a *connection\_status* value from the following list: - 0 = Not Connected - 1 = Connecting - **2 = In Sync** - 3 = Resync in Process - 4 = Config Change Restart Required | | Threshold value recommendation | This value should be 2 if everything is in sync. **Note**: At Typha startup, it is normal to have non-2 values, but it should stabilize at 2 after connections come up. | | Threshold breach symptoms | Typha will not receive updates from the relevant remote clusters. Connected clients will see stale or partial data from remote clusters. | | Threshold breach recommendations | Investigate Typha's logs where remote cluster connectivity events are logged. Ensure the networking between clusters is not experiencing issues. | | Priority level | Recommended. | ## Typha client metrics[​](#typha-client-metrics) ### Total connections accepted[​](#total-connections-accepted) | Total connections accepted | | | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `typha_connections_accepted` | | Example value | `typha_connections_accepted{endpoint="metrics-port", instance="10.0.1.20:9093", job="typha-metrics-svc", namespace="calico-system", pod="calico-typha-6c6cc9fcf7-csbdl", service="typha-metrics-svc"} 10` | | Explanation | Total number of connections accepted over time. This value always increases. | | Threshold value recommendation | A steady increase over time is normal. Counters rising after a Felix or Typha restart is also normal (as clients get rebalanced). Investigate connection counters that rise rapidly with no Felix or Typha restarts. | | Threshold breach symptoms | Counters rising when there are no Felix or Typha restarts, or no action that could cause restarts (an upgrade for example), could indicate unexpected Felix or Typha restarts or issues. | | Threshold breach recommendations | Check resource usage on Typha(s) and Kubernetes nodes. Increase resources if needed. | | Priority level | Optional. | ### Client connections actively streaming[​](#client-connections-actively-streaming) | Client connections actively streaming | | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Metric | `sum by (instance) (typha_connections_streaming)` | | Example value | `{instance="10.0.1.20:9093"} 10` `{instance="10.0.1.31:9093"} 5` | | Explanation | Current number of active connections that are "streaming" (have completed the handshake), to this Typha. After a connection has been Accepted (reported in the previous metric), there will be a handshake before the connection is deemed to be actively streaming. This indicates how many clients are connected to a Typha. The sum reflects per-cache metrics as well. | | Threshold value recommendation | Compare the value for **Total Connections Accepted** and **Client Connections Actively Streaming**. The fluctuation of these values should be in-sync with each other if Accepted Connections are turning into Actively Streamed connections. If there is a discrepancy , you should investigate. **Note**: As always, it is recommended to baseline the relationship between these two metrics to have a sense of what is normal. It is also worth noting that in smaller clusters, it is normal for Typha to be unbalanced. Typha can handle hundreds of connections so it is of no concern if all nodes in a 10-node cluster (for example) connect to the same Typha. | | Threshold breach symptoms | Felix is not getting updates from Typha. Calico Cloud network policies are out-of-sync. | | Threshold breach recommendations | Check Typha and Felix logs, and rollout restart Typha(s) if needed. | | Priority level | Recommended. | ### Rebalanced client connections[​](#rebalanced-client-connections) | Rebalanced client connections | | | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `rate(typha_connections_dropped{$_rate_interval})` | | Example value | `{endpoint="metrics-port", instance="10.0.1.20:9093", job="typha-metrics-svc", namespace="calico-system", pod="calico-typha-6c6cc9fcf7-csbdl", service="typha-metrics-svc"}` | | Explanation | Number of client connections dropped to rebalance and share the load across different Typhas. | | Threshold value recommendation | It is normal to see this value increasing sometimes. Investigate if connection dropped counters is rising constantly. If all Typhas are dropping connections because all Typhas believe they have too much load, this also warrants investigation. | | Threshold breach symptoms | Dropping connections is rate limited so it should not affect the cluster as a whole. Typha clients, like Felix, will get dropped sometimes (but not constantly), and could result in periodic delays to policy updates. | | Threshold breach recommendations | Ensure that the Kubernetes nodes have enough resources. | | Priority level | Optional. | ### 99 percentile client fall-behind[​](#99-percentile-client-fall-behind) | 99 percentile client fall-behind | | | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `max by (instance) (typha_client_latency_secs{quantile='0.99'})` | | Example value | `{instance="10.0.1.20:9093"} 0.1234` `{instance="10.0.1.31:9093"} 0.1234` | | Explanation | This metric measures how far behind Typha's client-handling threads are at reading updates.This metric will increase if: a) The client (e.g Felix) is slow or overloaded and cannot keep up with what Typha is sending or b) Typha is overloaded and it cannot keep up with writes to all its clients. This metric is a good indication of your cluster, Felix, and Typha health. | | Threshold value recommendation | It is normal for this to spike when new clients connect; they must download and process the snapshot, during which time they will fall slightly behind. Investigate of latency persists. | | Threshold breach symptoms | Typha clients receiving updates from Typha will be behind in time. Potential symptoms could include Calico Cloud network policies being out-of-sync. | | Threshold breach recommendations | Check Typha and Felix logs and resource usage. It is recommended to focus on Felix logs and resource usage first, as there is generally more overhead with Felix and thus more of a chance of overload. Rollout restart Typha(s) and calico-node(s) if needed. | | Priority level | Recommended. | ### 99 percentile client write latency[​](#99-percentile-client-write-latency) | 99 percentile client write latency | | | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `max by (instance) (typha_client_write_latency_secs)` | | Example value | `{instance="10.0.1.20:9093"} 0.007450815` | | Explanation | Time for Typha to write to a client's socket (for example, Felix). | | Threshold value recommendation | If the write latency is increasing, this indicates that a client (for example, Felix) is having an issue, or the network is having an issue. It is normal for intermittent spikes. Investigate any persistent latency. | | Threshold breach symptoms | Typha clients will lag behind in receiving updates that Typha is sending. Potential symptoms include Calico Cloud network policies being out-of-sync. | | Threshold breach recommendations | Check Felix logs and resource usage. | | Priority level | Recommended. | ### 99 percentile client ping latency[​](#99-percentile-client-ping-latency) | 99 percentile client ping latency | | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `max by (instance) (typha_ping_latency{quantile="0.99"})` | | Example value | `{instance="10.0.1.20:9093"} 0.034285331` | | Explanation | This metric tracks the round-trip-time from Typha to a client. How long it takes for Typha's clients to respond to pings over the Typha protocol. | | Threshold value recommendation | An increase in this metric above 1 second indicates that the clients, network or Typha are more heavily loaded. It is normal for intermittent spikes. Persistent latency above 1 second warrants investigation. | | Threshold breach symptoms | Typha clients could be behind in time on updates Typha is sending. Potential symptoms include Calico Cloud network policies being out-of-sync. | | Threshold breach recommendations | Check Typha and Felix logs and resource usage. It is recommended to focus on Felix logs and resource usage first, as there is generally more overhead with Felix and thus more of a chance of overload. Check if the node is overloaded and review/increase calico-node/Typha CPU requests if needed. If needed, rollout restart Typha(s) and calico-node(s). | | Priority level | Recommended. | ## Typha cache internals[​](#typha-cache-internals) ### 99 percentile breadcrumb size[​](#99-percentile-breadcrumb-size) | 99 percentile breadcrumb size | | | -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `max by (instance) (typha_breadcrumb_size{quantile="0.99"})` | | Explanation | Typha stores datastore changes as a series of blocks called breadcrumbs. Typha will store updates inside of these breadcrumbs (for example if a pod churned, this would be a single update). Typha can store multiple updates in a single breadcrumb with the default maximum size number being 100. | | Threshold value recommendation | Typha generating blocks of size 100 during start up is normal. Investigate if Typha is consistently generating blocks of size 90+, which can indicate Typha is overloaded. | | Threshold breach symptoms | Maintained block of sizes of 100 can indicate that Typha is falling behind on information and updates contained in the datastore. This will lead to Typha clients also falling behind (for example, Calico Cloud network policy object may not be current). | | Threshold breach recommendations | Check Typha logs and resource usage. Check if there is a lot of activity within the cluster that would cause Typha to send large breadcrumbs (for example, a huge amount of pod churn). If possible, reduce churn rate of resources on the cluster. | | Priority level | Recommended. | ### Non-blocking breadcrumbs fraction[​](#non-blocking-breadcrumbs-fraction) | Non-blocking breadcrumb fraction | | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `(sum by (instance) (rate(typha_breadcrumb_non_block{30s})))/((sum by (instance) (rate(typha_breadcrumb_non_block{30s})))+(sum by (instance) (rate(typha_breadcrumb_block{30s}))))` | | Example value | `{instance="10.0.1.20:9093"} NaN` | | Explanation | Typha stores datastore changes as a series of blocks called "breadcrumbs". Each client "follows the breadcrumbs" either by blocking and waiting, or skipping to the next one (non-blocking) if it is already available. **Non-blocking breadcrumb** **actions** indicates that Typha is constantly sending breadcrumbs to keep up with the datastore. **Blocking breadcrumb** **actions** indicate that Typha and the client have caught up, are up-to-date, and are waiting on the next breadcrumb. This metric will give a ratio between blocking and non-blocking actions that can indicate the health of Typha, its clients, and the cluster. | | Threshold value recommendation | As the load on Typha increases, the ratio of skip-ahead, non-blocking reads, increases. If it approaches 100% then Typha may be overloaded (since clients only do non-blocking reads when they're behind). | | Threshold breach symptoms | Consistent non-blocking breadcrumbs could indicate that Typha is falling behind on information and updates contained in the datastore. This will lead to Typha clients also being behind (for example, Calico Cloud network policy object may not be current). | | Threshold breach recommendations | Check Typha and Felix logs and resource usage. Check if there is a lot of activity within the cluster that would cause Typha to continuously send non-blocking breadcrumbs. | | Priority level | Recommended. | ### Datastore updates total[​](#datastore-updates-total) | Datastore updates total | | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | Metric | `sum by (instance) (rate(typha_updates_total{30s}))` | | Example value | `{instance="10.0.1.20:9093"} 0` | | Explanation | The rate of updates from the datastore(s). For example, updates to Pods/Nodes/Policies/etc. | | Threshold value recommendation | Intermittent spikes are expected. Constant updates indicates a very busy cluster (for example, lots of pod churn). | | Threshold breach symptoms | Constant updates could lead to overloaded Typhas whereTyphas clients could fall behind. | | Threshold breach recommendations | Ensure Typha has enough resources to handle a very dynamic cluster. | | Priority level | Optional. | ### Datastore update skipped (no-ops)[​](#datastore-update-skipped-no-ops) | Datastore update skipped (no-ops) | | | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `sum by (instance) (rate(typha_updates_skipped{30s}))` | | Example value | `{instance="10.0.1.20:9093"} 0` | | Explanation | The number of updates from the datastore that Typha detected were no-ops. For example, an update to a Kubernetes node resource that did not touch any values that is of interest to Calico Cloud. Such updates are not propagated to clients, which saves resources. | | Threshold value recommendation | N/A | | Threshold breach symptoms | N/A | | Threshold breach recommendations | N/A | | Priority level | Optional. | ## Typha snapshot details[​](#typha-snapshot-details) ### Snapshot send time[​](#snapshot-send-time) | Median snapshot send time | | | -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `max by (instance) (typha_client_snapshot_send_secs{quantile="0.5"})` | | Example value | `{instance="10.0.1.20:9093"} NaN` | | Explanation | The median time to stream the initial datastore snapshot to each client. It is useful to know the time it takes for a client to receive the data when it connects; it does not include time to process the data. | | Threshold value recommendation | Investigate if this value is moving towards 10s of seconds. | | Threshold breach symptoms | High values of this metric could indicate that newly-started clients are taking a long time to get the latest snapshot of the datastore, increasing the window of time where networking/policy updates are not being applied to the data plane during a restart/upgrade. Typha has a write timeout for writing the snapshot; if a client cannot receive the snapshot within that timeout, it is disconnected. Clients falling behind on information and updates contained in the datastore (for example, Calico Cloud network policy object may not be current). | | Threshold breach recommendations | Check Typha and calico-node logs and resource usage. Check for network congestion. Investigate why a particular calico-node is slow; it is likely on an overloaded node with insufficient CPU). | | Priority level | Optional. | ### Clients requiring grace period[​](#clients-requiring-grace-period) | Clients requiring grace period | | | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `sum by (instance) (typha_connections_grace_used)` | | Example value | `{instance="10.0.1.20:9093"} 0` | | Explanation | The number of Typhas with clients that required a grace period. After sending the snapshot to the client, Typha allows a grace period for the client to catch up to the most recent data. Typha sending the initial snapshot should take < 1 second, but the processing of the snapshot could take longer, so this grace period is there to allow the newly connected client to process the snapshot. | | Threshold value recommendation | If this metric is constantly increasing, it can indicate potential performance issues with Typha and clients. It can indicate that performance is being impacted and may warrant investigation. | | Threshold breach symptoms | High values of this metric could indicate clients falling behind on information and updates contained in the datastore (for example, Calico Cloud network policy object may not be current). | | Threshold breach recommendations | Check Typha and calico-node logs and resource usage. Check for network congestion, and determine the root cause. | | Priority level | Optional. | ### Max snapshot size (raw)[​](#max-snapshot-size-raw) | Max snapshot size (raw) | | | -------------------------------- | -------------------------------------------------------------- | | Metric | `max(typha_snapshot_raw_bytes)` | | Example value | `{} 557359` | | Explanation | The raw size in bytes of snapshots sent from Typha to clients. | | Threshold value recommendation | N/A | | Threshold breach symptoms | N/A | | Threshold breach recommendations | N/A | | Priority Level | Optional. | ### Max snapshot size (compressed)[​](#max-snapshot-size-compressed) | Max snapshot size (compressed) | | | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `max(typha_snapshot_compressed_bytes)` | | Example value | `{}134845` | | Explanation | The compressed size in bytes of snapshots sent from Typha to clients. | | Threshold value recommendation | This metric can be helpful for customers to estimate the bandwidth requirements for Felix to startup. For example, if the compressed snapshot size is 20MB in size on average, and 1000 Felix/calico-nodes start up, the bandwidth requirements could be estimated at 20GB between the pool of Typha and the set of Felixes across the network. | | Threshold breach symptoms | N/A | | Threshold breach recommendations | N/A | | Priority Level | Optional. | ## Policy metrics[​](#policy-metrics) > **SECONDARY:** The following policy metrics are a separate endpoint exposed by Felix that are used in the web console. They require special Prometheus configuration to scrape the metrics. For details, see [Policy metrics](https://docs.tigera.io/calico-cloud/operations/monitor/metrics/policy-metrics). ### Denied traffic[​](#denied-traffic) | Denied traffic | | | -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `calico_denied_packets calico_denied_bytes` | | Example value | `calico_denied_packets{endpoint="calico-metrics-port", instance="ip-10-0-1-30.ca-central-1.compute.internal", job="calico-node-metrics", namespace="calico-system", pod="calico-node-6pcqm", policy="default` | | Explanation | Number of packets or bytes that have been dropped by explicit or implicit deny rules. Note that you'll get one instance of `calico_denied_packets/bytes` for each policy rule that is denying traffic. For example: `calico_denied_packets{policy="tier1\|fv/policy1\|0\|deny\|-1",scrIP="10.245.13.133"}` | | Threshold value recommendation | The general rule of thumb is this metric should report zero at a stable state. Any deviation means that policy and traffic have diverged. Achieving a zero state depends on the stability and maturity of your cluster and policy. | | Threshold breach symptoms | Either unexpected traffic is being denied because of an attack (one example), or expected traffic is being denied because of a misconfiguration in a policy. | | Threshold breach recommendations | If this metric indicates that policy and traffic have diverged, the recommended steps are: Determine if an attack is causing the metric to spike, or if these flows should be allowed. If the flow should indeed be allowed, update the policy or a preceding policy to allow this traffic. | | Priority level | Recommended. | ### Traffic per rule[​](#traffic-per-rule) | Traffic per rule | | | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `cnx_policy_rule_bytes` `cnx_policy_rule_packets` | | Example value | `cnx_policy_rule_bytes{action="allow", endpoint="calico-metrics-port", instance="ip-10-0-1-20.ca-central-1.compute.internal", job="calico-node-metrics", namespace="calico-system", pod="calico-node-qzpkt", policy="es-kube-controller-access", rule_direction="egress", rule_index="1", service="calico-node-metrics", tier="allow-tigera", traffic_direction="inbound"}` | | Explanation | Number of bytes or packets handled by Calico Cloud network policy rules. | | Threshold value recommendation | This metric should usually be non-zero (unless expected). A zero value indicates the rule is not matching any packets, and could be surplus to requirements. | | Threshold breach symptoms | N/A | | Threshold breach recommendations | If this metrics consistently reports a zero value over an acceptable period of time, you can consider removing the policy rule. | | Priority Level | Optional. | ### Connections per policy rule[​](#connections-per-policy-rule) | Connections per policy rule | | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Metric | `cnx_policy_rule_connections` | | Example value | `cnx_policy_rule_connections{endpoint="calico-metrics-port", instance="ip-10-0-1-20.ca-central-1.compute.internal", job="calico-node-metrics", namespace="calico-system", pod="calico-node-qzpkt", policy="es-kube-controller-access", rule_direction="egress", rule_index="0", service="calico-node-metrics", tier="allow-tigera", traffic_direction="outbound"}` | | Explanation | Number connections handled by Calico Cloud policy rules. | | Threshold value recommendation | This metric is similar to *Traffic per Rule* but this deals more with **flow** monitoring. This metric should usually be non-zero. A zero value indicates that the rule is not matching any packets and could be surplus to requirements. | | Threshold breach symptoms | N/A | | Threshold breach recommendations | If this metrics consistently reports a zero value over an acceptable period of time, this policy rule can be considered for removal. | | Priority Level | Optional. | ## Felix cluster-state metrics[​](#felix-cluster-state-metrics) ### CPU usage[​](#cpu-usage-1) | CPU usage | | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `rate(process_cpu_seconds_total{30s}) * 100` | | Example value | `{endpoint="metrics-port", instance="10.0.1.20:9091", job="felix-metrics-svc", namespace="calico-system", pod="calico-node-qzpkt", service="felix-metrics-svc"}3.1197504199664072` | | Explanation | CPU in use by calico-node represented as a percentage of a core. | | Threshold value recommendation | A spike at startup is normal. It is recommended to first achieve a baseline and then monitor for any unexpected increases from this baseline. Investigate if maintained CPU usage goes above 90%. | | Threshold breach symptoms | Unexpected maintained CPU usage could cause Felix to fall behind and could cause delays to policy updates. | | Threshold breach recommendations | Check CPU usage on Kubernetes nodes. Increase resources if needed, rollout restart calico-node(s) if needed. | | Priority level | Recommended. | ### Memory usage[​](#memory-usage-1) | Memory usage | | | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `process_resident_memory_bytes` | | Example value | `process_resident_memory_bytes{endpoint="metrics-port", instance="10.0.1.20:9091", job="felix-metrics-svc", namespace="calico-system", pod="calico-node-qzpkt", service="felix-metrics-svc"} 98996224` | | Explanation | Amount of memory in use by calico-node. | | Threshold value recommendation | Recommended to achieve a baseline first, then monitor for any unexpected increases from this baseline. Investigate if maintained CPU usage goes above 90% of what is available from the underlying node. | | Threshold breach symptoms | Unexpected, maintained, memory usage could cause Felix to fall behind and could cause delays to policy updates. | | Threshold breach recommendations | Check memory usage on Kubernetes nodes. Increase resources if needed, rollout restart typha(s) if needed. | | Priority level | Recommended. | ### Active hosts on each endpoint[​](#active-hosts-on-each-endpoint) | Active hosts on each endpoint | | | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `felix_active_local_endpoints` | | Example value | `felix_active_local_endpoints{endpoint="metrics-port", instance="10.0.1.30:9091", job="felix-metrics-svc", namespace="calico-system", pod="calico-node-6pcqm", service="felix-metrics-svc"} 36` | | Explanation | Number of active **pod-networked** pods, and HEPs, on this node. | | Threshold value recommendation | Threshold relates to resource limits on the node for example kubelet's max pods setting. | | Threshold breach symptoms | Suggests Felix is getting out of sync. | | Threshold breach recommendations | Rolling restart calico-node and report issue to support. | | Priority level | Optional. | ### Active calico nodes[​](#active-calico-nodes) | Active calico nodes | | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | Metric | `max(felix_cluster_num_hosts)` | | Example value | `{} 3` | | Explanation | Total number of nodes in the cluster that have calico-node deployed and running. | | Threshold value recommendation | This value should be equal to the number of nodes in the cluster. If there are discrepancies, then calico-nodes on some nodes are having issues. | | Threshold breach symptoms | Calico Cloud network policies on affected nodes could be out-of-sync. | | Threshold breach recommendations | Check calico-node logs, rollout restart calico-node if needed. | | Priority level | Recommended. | ### Felix cluster policies[​](#felix-cluster-policies) | Felix cluster policies | | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Metric | `felix_cluster_num_policies` | | Example value | `felix_cluster_num_policies{endpoint="metrics-port", instance="10.0.1.20:9091", job="felix-metrics-svc", namespace="calico-system", pod="calico-node-qzpkt", service="felix-metrics-svc"} 58` | | Explanation | Total number of Calico Cloud network policies in the cluster. | | Threshold value recommendation | Because Calico Cloud is a distributed system, the number of policies should be generally consistent across all nodes. It is expected to have some skew between nodes for a short period of time while they sync, however they should never be out of sync for very long. | | Threshold breach symptoms | If nodes are out of sync for long time, calico-nodes may be having issues or experiencing resource contention. Check the Errors Plot to see if there are any iptables errors reported. | | Threshold breach recommendations | Redeploy calico-node if issues are seen, and increase resources if needed. | | Priority level | Optional. | ### Felix active local policies[​](#felix-active-local-policies) | Felix active local policies | | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `felix_active_local_policies` | | Example value | `felix_active_local_policies{endpoint="metrics-port", instance="10.0.1.30:9091", job="felix-metrics-svc", namespace="calico-system", pod="calico-node-6pcqm", service="felix-metrics-svc"} 44` | | Explanation | Total number of network policies deployed on per node basis. | | Threshold value recommendation | There is no hard limit to active policies. We can handle 1000+ active policies, but it impacts performance, especially if there's pod churn. The best solution is to optimize policies by combining multiple rules into one policy, and make sure that top-level policy selectors are being used. | | Threshold breach symptoms | N/A | | Threshold breach recommendations | Redeploy calico-node if issues are seen, and increase resources if needed. | | Priority level | Recommended. | ### Felix open FDS[​](#felix-open-fds) | Felix open FDS | | | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `sum by (pod) (process_open_fds{pod=~"calico-node.*"})` | | Example value | `{pod="calico-node-6pcqm"} 90` | | Explanation | Number of opened file descriptors per calico-node pod. | | Threshold value recommendation | Alert on this metric when it approaches the ulimit (as reported in `process_max_fds` value). You should not be anywhere near the maximum. | | Threshold breach symptoms | Felix may become unstable/crash or fail to apply updates as it should. These failures and issues are logged. | | Threshold breach recommendations | Check Felix logs, redeploy calico-node if you see log issues, and increase `max_fds value` if possible. | | Priority Level | Optional. | ### Felix max FDS[​](#felix-max-fds) | Felix max FDS | | | -------------------------------- | ---------------------------------------------------------------------- | | Metric | `sum by (pod) (process_max_fds{pod=~"calico-node.*"})` | | Example value | `{pod="calico-node-qzpkt"} 1048576` | | Explanation | Maximum number of opened file descriptors allowed per calico-node pod. | | Threshold value recommendation | N/A | | Threshold breach symptoms | N/A | | Threshold breach recommendations | N/A | | Priority level | Optional. | ### Felix resync started[​](#felix-resync-started) | Felix resync started | | | -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `sum(rate(felix_resyncs_started{5m}))` | | Explanation | This is the number of times that Typha has reported to Felix that it is re-connecting with the datastore. | | Threshold value recommendation | Occasional resyncs are normal. Investigate resync counters that rapidly rise. | | Threshold breach symptoms | Typha pods may be having issues or experiencing resource contention. Some calico-nodes that are paired with Typha pods experiencing issues will not be able to sync with the datastore. | | Threshold breach recommendations | Investigate the root cause to avoid redeploying Typha (which can be very disruptive). Check resource contention and network connectivity from Typha to the datastore to see if Typha is working fine or if the API server is overloaded. | | Priority level | Recommended. | ### Felix dropped logs[​](#felix-dropped-logs) | Felix dropped logs | | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Metric | `felix_logs_dropped` | | Example value | `felix_logs_dropped{endpoint="metrics-port", instance="10.0.1.20:9091", job="felix-metrics-svc", namespace="calico-system", pod="calico-node-qzpkt", service="felix-metrics-svc"} 0` | | Explanation | The number of logs Felix has dropped. Note that this metric does not count flow-logs; it counts logs to stdout. | | Threshold value recommendation | Occasional drops are normal. Investigate if drop counters rapidly rise. | | Threshold breach symptoms | Felix will drop logs if it cannot keep up with writing them out. These are ordinary code logs, not flow logs. Calico-node may be under resource constraints. | | Threshold breach recommendations | Check CPU usage on calico-nodes and Kubernetes nodes. Increase resources if needed, and rollout restart calico-node(s) if needed. | | Priority level | Optional. | ## Felix error metrics[​](#felix-error-metrics) ### IPset errors[​](#ipset-errors) | IPset errors | | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `sum(rate(felix_ipset_errors{5m}))` | | Example value | `{} 0` | | Explanation | Number of ipset creation, modification, and deletion command failures. This metric reports how many times the ipset command has failed when Felix tried to run it. An error can occur when Felix sends bad ipset command data, or the kernel throws an error (potentially because it was too busy to handle this request at that time). | | Threshold value recommendation | Occasional errors are normal. Investigate error counters that rapidly rise. | | Threshold breach symptoms | Calico Cloud network policies may not scope all endpoints in network policy rules. Cluster nodes may be under resource contention, which may result in other \_error and \_seconds metrics rising. Repeated errors could mean some persistent problem (for example, some other process has created an IP set with that name, which is incompatible). | | Threshold breach recommendations | See the **Errors Plot graph** to determine if the scope is cluster-wide or node-local. Check calico-node logs. Check resource usage and contention on Kubernetes nodes and calico-nodes. Add nodes/resources if needed. If resource contention is not seen, restart calico-node(s) and monitor. Ensure that other process using IPtables are not blocking Calico Cloud network policy management. | | Priority level | Optional. | ### Iptables restore errors[​](#iptables-restore-errors) | Iptables restore errors | | | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `sum(rate(felix_iptables_restore_errors{5m}))` | | Explanation | The number of iptables-restore errors over five minutes. The iptables-restore command is used when Calico Cloud makes a change to iptables. For example, a new WEP or HEP is created, changes to a WEP or HEP or a change to a policy that affects a WEP or HEP. | | Threshold value recommendation | Occasional errors are normal. Investigate error counters that rapidly rise. | | Threshold breach symptoms | Calico Cloud network policies are not up to date. Cluster nodes may be under resource contention, which may result in other \_error and \_seconds metrics rising. | | Threshold breach recommendations | See the Errors Plot graph to determine if the scope is cluster-wide or node-local. Check calico-node logs. Check resource usage and contention on Kubernetes nodes and calico-nodes. Add nodes/resources if needed. If no resource contention is seen, restart calico-node and monitor. | | Priority level | Optional. | ### Iptables save errors[​](#iptables-save-errors) | Iptables save errors | | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Metric | `sum(rate(felix_iptables_save_errors{5m}))` | | Example value | `{} 0` | | Explanation | Number of iptables-save errors. The iptables-save command is run before every iptables-restore command so that Calico Cloud has the current state of iptables. | | Threshold value recommendation | Occasional errors are normal. Investigate error counters that rapidly rise. | | Threshold breach symptoms | Calico Cloud network policies are not up to date. Cluster nodes may be under resource contention, which may result in other \_error and \_seconds metrics rising. Repeated errors could mean some persistent problem (for example, some other process has creating iptables rules that Calico Cloud cannot decode with the version of iptables-save in use). | | Threshold breach recommendations | See the Errors Plot graph to determine if the scope is cluster-wide or node-local. Check calico-node logs. Check resource usage and contention on Kubernetes nodes and calico-nodes. Add nodes/resources if needed. If no resource contention is seen, restart calico-node and monitor. | | Priority level | Optional. | ### Felix log errors[​](#felix-log-errors) | Felix log errors | | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `sum(rate(felix_log_errors{5m}))` | | Example value | `{} 0` | | Explanation | The number of times Felix fails to write out a log because the log buffer is full. | | Threshold value recommendation | Occasional errors are normal. Investigate error counters that rapidly rise. | | Threshold breach symptoms | Calico-node may be under resource contention, which may result in other **\_error** and **\_seconds** metrics rising. | | Threshold breach recommendations | See the **Errors Plot graph** to determine if the scope is cluster-wide or node-local. Check resource usage and contention on Kubernetes nodes and calico-nodes. Add nodes/resources if needed. If no resource contention is seen, restart calico-node and monitor. | | Priority level | Optional. | ### Monitor Felix metrics using a graph[​](#monitor-felix-metrics-using-a-graph) | Errors plot graph | | | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `rate(felix_ipset_errors{5m}) \|\| rate(felix_iptables_restore_errors[5m]) \|\| rate(felix_iptables_save_errors[5m]) \|\| rate(felix_log_errors{5m})` | | Example value | `{endpoint="metrics-port", instance="10.0.1.20:9091", job="felix-metrics-svc", namespace="calico-system", pod="calico-node-qzpkt", service="felix-metrics-svc"} 0` | | Explanation | Checks if there have been any iptables-save, iptables-restore, or ipset command errors in the past five minutes. Keeps track of what node is reporting which error. | | Threshold value recommendation | Occasional errors are normal. Investigate error counters that rapidly rise. For this specific metric it is worth focusing on the metric that is spiking, and referencing that metric information. | | Threshold breach symptoms | Dependent on the specific metric that is logging errors. | | Threshold breach recommendations | If more than one metric is rising, check if all rising metrics are related to a specific calico-node. If this is the case, then the issue is local to that calico-node. Check calico-node logs. Check resource usage for the node and calico-node pod. If more than one metric is rising rapidly across all calico-nodes, then it is a cluster-wide issue and cluster health must be checked. Check cluster resource usage, cluster networking/infrastructure health, and restart calico-nodes and calico-typha pods. | | Priority level | Recommended. | ## Felix time-based metrics[​](#felix-time-based-metrics) ### Data plane apply time quantile 0.5/0.9/0.99[​](#data-plane-apply-time-quantile-0509099) | Data plane apply time quantile 0.5/0.9/0.99 | | | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `felix_int_dataplane_apply_time_seconds{quantile="0.5"}` `felix_int_dataplane_apply_time_seconds{quantile="0.9"}` `felix_int_dataplane_apply_time_seconds{quantile="0.99"}` | | Example value | `felix_int_dataplane_apply_time_seconds{quantile="0.5"}:felix_int_dataplane_apply_time_seconds{endpoint="metrics-port", instance="10.0.1.30:9091", job="felix-metrics-svc", namespace="calico-system", pod="calico-node-6pcqm", quantile="0.5", service="felix-metrics-svc"} 0.020859218` | | Explanation | Time in seconds that it took to apply a data plane update ,viewed at the median, 90th percentile, and 99th percentile. | | Threshold value recommendation | Thresholds will vary depending on cluster size and rate of churn. It is recommended that a baseline be set to determine a normal threshold value. In the field we have seen >10s in extremely high-scale clusters with 100k+ endpoints and lots of policy/Kubernetes services. | | Threshold breach symptoms | Large time-to-apply values will cause a delay between Calico Cloud network policy commits and enforcement in the data plane. This is dependent on how Calico Cloud waiting for kube-proxy to release the iptables lock, which is influenced by the number of services in use. | | Threshold breach recommendations | Increase cluster resources, and reduce the number of Kubernetes services if possible. | | Priority level | Recommended. | ### Felix route table list seconds quantile 0.5/0.9/0.99[​](#felix-route-table-list-seconds-quantile-0509099) | Felix route table list seconds quantile 0.5/0.9/0.99 | | | ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `felix_route_table_list_seconds{quantile="0.5"}` `felix_route_table_list_seconds{quantile="0.9"}` `felix_route_table_list_seconds{quantile="0.99"}` | | Example value | `felix_route_table_list_seconds{quantile="0.5"}:``felix_route_table_list_seconds{endpoint="metrics-port",``instance="10.0.1.30:9091",job="felix-metrics-svc",``namespace="calico-system", pod="calico-node-6pcqm",``quantile="0.5", service="felix-metrics-svc"} 0.000860426` | | Explanation | Time to list all the interfaces during a resync, viewed at the median, 90th percentile and 99th percentile. | | Threshold value recommendation | Thresholds will vary depending on the number of cali interfaces per node. It is recommended that a baseline be set to determine a normal threshold value. | | Threshold breach symptoms | High values indicate high CPU usage in felix and slow data plane updates. | | Threshold breach recommendations | Increase cluster resources. Reduce the number of cali interfaces per node where possible. | | Priority level | Optional. | ### Felix graph update time quantile 0.5/0.9/0/99[​](#felix-graph-update-time-quantile-0509099) | Felix graph update time seconds quantile 0.5/0.9/0.99 | | | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Metric | `felix_calc_graph_update_time_seconds{quantile="0.5"}` `felix_calc_graph_update_time_seconds{quantile="0.9"}` `felix_calc_graph_update_time_seconds{quantile="0.99"}` | | Example value | `felix_calc_graph_update_time_seconds{quantile="0.5"}:``felix_calc_graph_update_time_seconds{endpoint="metrics-port",``instance="10.0.1.30:9091", job="felix-metrics-svc",``namespace="calico-system", pod="calico-node-6pcqm",``quantile="0.5", service="felix-metrics-svc"} 0.00007129` | | Explanation | This metric reports the time taken to update the calculation graph for each datastore on an update call, viewed at the median, 90th percentile and 99th percentile. The calculation graph is the Felix component that takes all the policies/workload endpoints/host endpoints information that it has received from Typha, and distills it down to data plane updates that are relevant for this node. | | Threshold value recommendation | After *start of day* (where we will typically get a large update), then values should be sub 1 second (with occasional blips to 1+ seconds). Should be measured in milliseconds with the occasional blip to a second or two. Investigate if the result is constantly in values of seconds. | | Threshold breach symptoms | High values indicate high CPU usage in felix and slow data plane updates. | | Threshold breach recommendations | Increase cluster resources. Check calico-node logs. Rollout restart calico-node(s) if needed. | | Priority level | Recommended. | ### Bring your own Prometheus ## Big picture[​](#big-picture) Scrape Calico Cloud metrics for Bring Your Own (BYO) Prometheus. ## Value[​](#value) Calico Cloud uses the Prometheus monitoring tool to scrape metrics from instrumented jobs, and displays time-series data in a visualizer such as Grafana. You can scrape the following time-series metrics for Calico Cloud components to your own Prometheus: - elasticsearch - fluentd - calico-node - kube-controllers - felix - typha (not enabled by default) ## Before you begin[​](#before-you-begin) **Supported** For the supported version of Prometheus in this release, see the [Release Notes](https://docs.tigera.io/calico-cloud/release-notes/) (`coreos-prometheus`). ## How to[​](#how-to) - [Scrape all enabled metrics](#scrape-all-enabled-metrics) - [Scrape metrics from specific components directly](#scrape-metrics-from-specific-components-directly) - [Verify BYO Prometheus](#verify-byo-prometheus) - [Create policy to secure traffic between pods](#create-policy-to-secure-traffic-between-pods) - [Troubleshooting](#troubleshooting) ### Scrape all enabled metrics[​](#scrape-all-enabled-metrics) In this section we create a service monitor that scrapes all enabled metrics. To enable metrics that are not enabled by default, please consult the [next section](#scrape-metrics-from-specific-components). The following example shows a Prometheus server installed in namespace "external-prometheus" with a `serviceMonitorSelector` that selects all service monitors with the label `k8s-app=tigera-external-prometheus`. 1. Save the following configuration in a file called `monitor.yaml`. ```yaml apiVersion: operator.tigera.io/v1 kind: Monitor metadata: name: tigera-secure spec: externalPrometheus: namespace: external-prometheus serviceMonitor: labels: k8s-app: tigera-external-prometheus ``` For a list of all configuration options, see the [Installation API reference](https://docs.tigera.io/calico-cloud/reference/installation/api). 2. Apply the manifest to your cluster. ```bash kubectl apply -f monitor.yaml ``` 3. Verify that the new configuration has been added to your cluster ```bash export NS=external-prometheus kubectl get servicemonitor -n $NS tigera-external-prometheus kubectl get serviceaccount -n $NS tigera-external-prometheus kubectl get secret -n $NS tigera-external-prometheus kubectl get clusterrole tigera-external-prometheus kubectl get clusterrolebinding tigera-external-prometheus ``` That's it. You should be seeing the new metrics show up in your Prometheus instance within a minute. For more information on verifying metrics, see the section, [Verify BYO Prometheus](#verify-byo-prometheus). ### Scrape metrics from specific components directly[​](#scrape-metrics-from-specific-components-directly) We recommend the previous section for scraping all enabled metrics. Read on if you wish to scrape metrics from specific components directly using mTLS, or if you wish to enable metrics that are disabled by default. **Tab: elasticsearch** **Configure TLS certificates** 1. Copy the required secret and configmap to your namespace. 2. Save the manifest of the required TLS secret and CA configmap. ```bash kubectl get secret calico-node-prometheus-client-tls -n tigera-prometheus -o yaml > calico-node-prometheus-client-tls.yaml ``` ```bash kubectl get configmap -n tigera-prometheus tigera-ca-bundle -o yaml > tigera-ca-bundle.yaml ``` 3. Edit `calico-node-prometheus-client-tls.yaml` and `tigera-ca-bundle.yaml` by changing the namespace to the namespace where your prometheus instance is running. 4. Apply the manifests to your cluster. ```bash kubectl apply -f calico-node-prometheus-client-tls.yaml ``` ```bash kubectl apply -f tigera-ca-bundle.yaml ``` **Create the service monitor** Apply the ServiceMonitor to the namespace where Prometheus is running. ```bash export NAMESPACE= ``` ```bash kubectl apply -f https://downloads.tigera.io/ee/v3.22.0-2.0/manifests/prometheus/elasticsearch-metrics-service-monitor.yaml -n $NAMESPACE ``` The .yamls have no namespace defined so when you apply `kubectl`, it is applied in the $NAMESPACE. **Tab: fluentd** **Configure TLS certificates** 1. Copy the required secret and configmap to your namespace. 2. Save the manifest of the required TLS secret and CA configmap. ```bash kubectl get secret calico-node-prometheus-client-tls -n tigera-prometheus -o yaml > calico-node-prometheus-client-tls.yaml ``` ```bash kubectl get configmap -n tigera-prometheus tigera-ca-bundle -o yaml > tigera-ca-bundle.yaml ``` 3. Edit `calico-node-prometheus-client-tls.yaml` and `tigera-ca-bundle.yaml` and change the namespace to the namespace where your prometheus instance is running. 4. Apply the manifests to your cluster. ```bash kubectl apply -f calico-node-prometheus-client-tls.yaml ``` ```bash kubectl apply -f tigera-ca-bundle.yaml ``` **Create the service monitor** Apply the ServiceMonitor to the namespace where Prometheus is running. ```bash export NAMESPACE= ``` ```bash kubectl apply -f https://downloads.tigera.io/ee/v3.22.0-2.0/manifests/prometheus/fluentd-metrics-service-monitor.yaml -n $NAMESPACE ``` The .yamls have no namespace defined so when you apply `kubectl`, it is applied in the $NAMESPACE. **Tab: calico node** **Configure TLS certificates** 1. Copy the required secret and configmap to your namespace. 2. Save the manifest of the required TLS secret and CA configmap. ```bash kubectl get secret calico-node-prometheus-client-tls -n tigera-prometheus -o yaml > calico-node-prometheus-client-tls.yaml ``` ```bash kubectl get configmap -n tigera-prometheus tigera-ca-bundle -o yaml > tigera-ca-bundle.yaml ``` 3. Edit `calico-node-prometheus-client-tls.yaml` and `tigera-ca-bundle.yaml` by changing the namespace to the namespace where your prometheus instance is running. 4. Apply the manifests to your cluster. ```bash kubectl apply -f calico-node-prometheus-client-tls.yaml ``` ```bash kubectl apply -f tigera-ca-bundle.yaml ``` **Create the service monitor** Apply the ServiceMonitor to the namespace where Prometheus is running. ```bash export NAMESPACE= ``` ```bash kubectl apply -f https://downloads.tigera.io/ee/v3.22.0-2.0/manifests/prometheus/calico-node-monitor-service-monitor.yaml -n $NAMESPACE ``` The .yamls have no namespace defined so when you apply `kubectl`, it is applied in $NAMESPACE. **Tab: kube-controllers** **Configure TLS certificates** 1. Copy the required secret and configmap to your namespace. 2. Save the manifest of the required TLS secret and CA configmap. ```bash kubectl get secret calico-node-prometheus-client-tls -n tigera-prometheus -o yaml > calico-node-prometheus-client-tls.yaml ``` ```bash kubectl get configmap -n tigera-prometheus tigera-ca-bundle -o yaml > tigera-ca-bundle.yaml ``` 3. Edit `calico-node-prometheus-client-tls.yaml` and `tigera-ca-bundle.yaml` by changing the namespace to the namespace where your prometheus instance is running. 4. Apply the manifests to your cluster. ```bash kubectl apply -f calico-node-prometheus-client-tls.yaml ``` ```bash kubectl apply -f tigera-ca-bundle.yaml ``` **Create the service monitor** Apply the ServiceMonitor to the namespace where Prometheus is running. ```bash export NAMESPACE= ``` ```bash kubectl apply -f https://downloads.tigera.io/ee/v3.22.0-2.0/manifests/prometheus/kube-controller-metrics-service-monitor.yaml -n $NAMESPACE ``` The .yamls have no namespace defined so when you apply `kubectl`, it is applied in the $NAMESPACE. **Tab: Felix** **Enable metrics** Felix metrics are not enabled by default. By default, Felix uses **port 9091 TCP** to publish metrics. Use the following command to enable Felix metrics. ```bash kubectl patch felixconfiguration default --type merge --patch '{"spec":{"prometheusMetricsEnabled": true}}' ``` You should see a result similar to: ```text felixconfiguration.projectcalico.org/default patched ``` For all Felix configuration values, see [Felix configuration](https://docs.tigera.io/calico-cloud/reference/component-resources/node/felix/configuration). For all Prometheus Felix configuration values, see [Felix Prometheus](https://docs.tigera.io/calico-cloud/reference/component-resources/node/felix/prometheus). **For Windows nodes, create a service to expose Felix metrics** If you're running Calico Enterprise for Windows, you must create a service to expose Felix metrics for Windows nodes: ```bash kubectl apply -f - < ``` ```bash kubectl apply -f https://downloads.tigera.io/ee/v3.22.0-2.0/manifests/prometheus/felix-metrics-service-monitor.yaml -n $NAMESPACE ``` The .yamls have no namespace defined so when you apply `kubectl`, it is applied in the $NAMESPACE. **Tab: Typha** **Enable metrics** Typha metrics are not enabled by default. By default, Typha uses **port 9091** TCP to publish metrics. However, if Calico Cloud is installed using the Amazon yaml file, this port will be 9093 because it is set manually using the **TYPHA\_PROMETHEUSMETRICSPORT** environment variable. Use the following command to enable Typha metrics. ```bash kubectl patch installation default --type=merge -p '{"spec": {"typhaMetricsPort":9093}}' ``` You should see a result similar to: ```bash installation.operator.tigera.io/default patched ``` **Create the service monitor** Apply the ServiceMonitor to the namespace where Prometheus is running. ```bash export NAMESPACE= ``` ```bash kubectl apply -f https://downloads.tigera.io/ee/v3.22.0-2.0/manifests/prometheus/typha-metrics-service-monitor.yaml -n $NAMESPACE ``` The .yamls have no namespace defined so when you apply `kubectl`, it is applied in the $NAMESPACE. ### Verify BYO Prometheus[​](#verify-byo-prometheus) 1. Access the Prometheus dashboard using the port-forwarding feature. ```bash kubectl port-forward pod/byo-prometheus-pod 9090:9090 -n $NAMESPACE ``` 2. Browse to the Prometheus dashboard: [http://localhost:9090](http://localhost:9090). 3. In the Expression text box, enter your metric name and click the **Execute** button. The Console table is populated with all of your nodes with the number of endpoints. ### Troubleshooting[​](#troubleshooting) This section is applicable only if you experience issues with mTLS after following the [Scrape metrics from specific components directly](#scrape-metrics-from-specific-components) section. 1. Use the following command to retrieve the tls.key and tls.cert. ```bash export NAMESPACE= ``` ```bash kubectl get secret -n $NAMESPACE calico-node-prometheus-client-tls -o yaml ``` 2. Save the tls.key and tls.cert content into key and cert after base64 decode. ```bash $:tls_key= $:echo $tls_key|base64 -d >key.pem $:tls_cert= $:echo $cert|base64 -d>cert.pem ``` 3. Get the ca-bundle certificate using this command: ```bash kubectl get cm -n $NAMESPACE tigera-ca-bundle -o yaml ``` 4. Open a new file (bundle.pem) in your favorite editor, and paste the content from "BEGIN CERTIFICATE" to "END CERTIFICATE". 5. Port-forward the prometheus pods and run this command with the forwarded port. ```bash curl --cacert bundle.pem --key key.pem --cert cert.pem https://localhost:8080/metrics ``` You should be able to see the metrics. ### Create policy to secure traffic between pods[​](#create-policy-to-secure-traffic-between-pods) To support zero trust, we recommend that you create Calico Cloud network policy to allow the traffic between BYO Prometheus pods, and the respective metrics pods. For samples of ingress and egress policies, see [Get started with Calico network policy](https://docs.tigera.io/calico-cloud/network-policy/beginners/calico-network-policy). ### Configure Prometheus ## Updating denied packets rules[​](#updating-denied-packets-rules) This is an example of how to modify the sample rule created by the sample manifest. The process of updating rules is the same as for user created rules (documented below). - Save the current alert rule: ```bash kubectl -n tigera-prometheus get prometheusrule -o yaml > calico-prometheus-alert-rule-dp.yaml ``` - Make necessary edits to the alerting rules then apply the updated manifest. ```bash kubectl apply -f calico-prometheus-alert-rule-dp.yaml ``` Your changes should be applied in a few seconds by the prometheus-config-reloader container inside the prometheus pod launched by the prometheus-operator (usually named `prometheus-`). As an example, the range query in this Manifest is 10 seconds. ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: calico-prometheus-dp-rate namespace: tigera-prometheus labels: role: tigera-prometheus-rules prometheus: calico-node-prometheus spec: groups: - name: calico.rules rules: - alert: DeniedPacketsRate expr: rate(calico_denied_packets[10s]) > 50 labels: severity: critical annotations: summary: 'Instance {{$labels.instance}} - Large rate of packets denied' description: '{{$labels.instance}} with calico-node pod {{$labels.pod}} has been denying packets at a fast rate {{$labels.sourceIp}} by policy {{$labels.policy}}.' ``` To update this alerting rule, to say, execute the query with a range of 20 seconds modify the manifest to this: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: calico-prometheus-dp-rate namespace: tigera-prometheus labels: role: tigera-prometheus-rules prometheus: calico-node-prometheus spec: groups: - name: calico.rules rules: - alert: DeniedPacketsRate expr: rate(calico_denied_packets[20s]) > 50 labels: severity: critical annotations: summary: 'Instance {{$labels.instance}} - Large rate of packets denied' description: '{{$labels.instance}} with calico-node pod {{$labels.pod}} has been denying packets at a fast rate {{$labels.sourceIp}} by policy {{$labels.policy}}.' ``` ## Creating a new alerting rule[​](#creating-a-new-alerting-rule) Creating a new alerting rule is straightforward once you figure out what you want your rule to look for. Check [alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) and [Queries](https://prometheus.io/docs/querying/examples/) for more information. ### New alerting rule for monitoring Calico node[​](#new-alerting-rule-for-monitoring-calico-node) To add the new alerting rule to our Prometheus instance, define a PrometheusRule manifest in the `tigera-prometheus` namespace with the labels `role: tigera-prometheus-rules` and `prometheus: calico-node-prometheus`. The labels should match the labels defined by the `ruleSelector` field of the Prometheus manifest. As an example, to fire a alert when a calico-node instance has been down for more than 5 minutes, save the following to a file, say `calico-node-down-alert.yaml`. ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: calico-prometheus-calico-node-down namespace: tigera-prometheus labels: role: tigera-prometheus-rules prometheus: calico-node-prometheus spec: groups: - name: calico.rules rules: - alert: CalicoNodeInstanceDown expr: up == 0 for: 5m labels: severity: warning annotations: summary: 'Instance {{$labels.instance}} Pod: {{$labels.pod}} is down' description: '{{$labels.instance}} of job {{$labels.job}} has been down for more than 5 minutes' ``` Then create/apply this manifest in kubernetes. ```bash kubectl apply -f calico-node-down-alert.yaml ``` Your changes should be applied in a few seconds by the prometheus-config-reloader container inside the prometheus pod launched by the prometheus-operator (usually named `prometheus-`). ### New alerting rule for monitoring BGP peers[​](#new-alerting-rule-for-monitoring-bgp-peers) Let’s look at an example of a new alerting rule to our Prometheus instance with respect to monitoring BGP peering health. Define a PrometheusRule manifest in the tigera-prometheus namespace with the labels `role: tigera-prometheus-rules` and `prometheus: calico-node-prometheus`. The labels should match the labels defined by the `ruleSelector` field of the Prometheus manifest. As an example, to fire an alert when the number of peering connections with a status other than “Established” is increasing at a non-zero rate in the cluster (over the last 5 minutes), save the following to a file, say `tigera-peer-status-not-established.yaml`. ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels: prometheus: calico-node-prometheus role: tigera-prometheus-rules name: tigera-prometheus-peer-status-not-established namespace: tigera-prometheus spec: groups: - name: calico.rules rules: - alert: CalicoNodePeerStatusNotEstablished annotations: description: '{{$labels.instance}} has at least one peer connection that is no longer up.' summary: Instance {{$labels.instance}} has peer connection that is no longer up expr: rate(bgp_peers{status!~"Established"}[5m]) > 0 labels: severity: critical ``` Then create/apply this manifest in kubernetes. ```bash kubectl apply -f tigera-peer-status-not-established.yaml ``` Your changes should be applied in a few seconds by the prometheus-config-reloader container inside the prometheus pod launched by the prometheus-operator (usually named `prometheus-`). ## Additional Alerting Rules[​](#additional-alerting-rules) The Alerting Rules installed by the Calico Cloud install manifest is a simple one that fires an alert when the rate of denied packets denied by a policy on a node from a particular Source IP exceeds a certain packets per second threshold. The Prometheus query used for this (ignoring the threshold value 20) is: ```text rate(calico_denied_packets[10s]) ``` and this query will return results something along the lines of: ```text {endpoint="calico-metrics-port",instance="10.240.0.81:9081",job="calico-node-metrics",namespace="kube-system",pod="calico-node-hn0kl",policy="profile/k8s_ns.test/0/deny",service="calico-node-metrics",srcIP="192.168.167.129"} 0.6 {endpoint="calico-metrics-port",instance="10.240.0.84:9081",job="calico-node-metrics",namespace="kube-system",pod="calico-node-97m3g",policy="profile/k8s_ns.test/0/deny",service="calico-node-metrics",srcIP="192.168.167.175"} 0.2 {endpoint="calico-metrics-port",instance="10.240.0.84:9081",job="calico-node-metrics",namespace="kube-system",pod="calico-node-97m3g",policy="profile/k8s_ns.test/0/deny",service="calico-node-metrics",srcIP="192.168.252.157"} 0.4 {endpoint="calico-metrics-port",instance="10.240.0.81:9081",job="calico-node-metrics",namespace="kube-system",pod="calico-node-hn0kl",policy="profile/k8s_ns.test/0/deny",service="calico-node-metrics",srcIP="192.168.167.175"} 1 {endpoint="calico-metrics-port",instance="10.240.0.84:9081",job="calico-node-metrics",namespace="kube-system",pod="calico-node-97m3g",policy="profile/k8s_ns.test/0/deny",service="calico-node-metrics",srcIP="192.168.167.129"} 0.4 {endpoint="calico-metrics-port",instance="10.240.0.81:9081",job="calico-node-metrics",namespace="kube-system",pod="calico-node-hn0kl",policy="profile/k8s_ns.test/0/deny",service="calico-node-metrics",srcIP="192.168.167.159"} 0.4 {endpoint="calico-metrics-port",instance="10.240.0.81:9081",job="calico-node-metrics",namespace="kube-system",pod="calico-node-hn0kl",policy="profile/k8s_ns.test/0/deny",service="calico-node-metrics",srcIP="192.168.252.175"} 0.4 {endpoint="calico-metrics-port",instance="10.240.0.84:9081",job="calico-node-metrics",namespace="kube-system",pod="calico-node-97m3g",policy="profile/k8s_ns.test/0/deny",service="calico-node-metrics",srcIP="192.168.252.175"} 0.6 {endpoint="calico-metrics-port",instance="10.240.0.81:9081",job="calico-node-metrics",namespace="kube-system",pod="calico-node-hn0kl",policy="profile/k8s_ns.test/0/deny",service="calico-node-metrics",srcIP="192.168.252.157"} 0.6 {endpoint="calico-metrics-port",instance="10.240.0.84:9081",job="calico-node-metrics",namespace="kube-system",pod="calico-node-97m3g",policy="profile/k8s_ns.test/0/deny",service="calico-node-metrics",srcIP="192.168.167.159"} 0.6 ``` We can modify this query to find out all packets dropped by different policies on every node. ```text (sum by (instance,policy) (rate(calico_denied_packets[10s]))) ``` This query will aggregate the results from all different Source IPs, and preserve the `policy` and `instance` labels. Note that the `instance` label represents the calico node's IP Address and `PrometheusReporterPort`. This query will return results like so: ```text {instance="10.240.0.84:9081",policy="profile/k8s_ns.test/0/deny"} 2 {instance="10.240.0.81:9081",policy="profile/k8s_ns.test/0/deny"} 2.8 ``` To include the pod name in these results, add the label `pod` to the labels listed in the `by` expression like so: ```text (sum by (instance,pod,policy) (rate(calico_denied_packets[10s]))) ``` which will return the following results: ```text {instance="10.240.0.84:9081",pod="calico-node-97m3g",policy="profile/k8s_ns.test/0/deny"} 2 {instance="10.240.0.81:9081",pod="calico-node-hn0kl",policy="profile/k8s_ns.test/0/deny"} 2.8 ``` An interesting use case is when a rogue Pod is using tools such as nmap to scan a subnet for open ports. To do this, we have to execute a query that will aggregate across all policies on all instances while preserving the source IP address. This can be done using this query: ```text (sum by (srcIP) (rate(calico_denied_packets[10s]))) ``` which will return results, different source IP address: ```text {srcIP="192.168.167.159"} 1.0000000000000002 {srcIP="192.168.167.129"} 1.2000000000000002 {srcIP="192.168.252.175"} 1.4000000000000001 {srcIP="192.168.167.175"} 0.4 {srcIP="192.168.252.157"} 1.0000000000000002 ``` To use these queries as Alerting Rules, follow the instructions defined in the [Creating a new Alerting Rule](#creating-a-new-alerting-rule) section and create a ConfigMap with the appropriate query. ## Updating the scrape interval[​](#updating-the-scrape-interval) You may wish to modify the scrape interval (time between Prometheus polling each node for new denied packet information). Increasing the interval reduces load on Prometheus and the amount of storage required, but decreases the detail of the collected metrics. The scrape interval of endpoints (calico-node in our case) is defined as part of the ServiceMonitor manifest. To change the interval: - Save the current ServiceMonitor manifest: ```bash kubectl -n tigera-prometheus get servicemonitor calico-node-monitor -o yaml > calico-node-monitor.yaml ``` - Update the `interval` field under `endpoints` to desired settings and apply the updated manifest. ```bash kubectl apply -f calico-node-monitor.yaml ``` Your changes should be applied in a few seconds by the prometheus-config-reloader container inside the prometheus pod launched by the prometheus-operator (usually named `prometheus-`). As an example on what to update, the interval in this ServiceMonitor manifest is 5 seconds (`5s`). ```yaml apiVersion: monitoring.coreos.com/v1alpha1 kind: ServiceMonitor metadata: name: calico-node-monitor namespace: tigera-prometheus labels: team: network-operators spec: selector: matchLabels: k8s-app: calico-node namespaceSelector: matchNames: - kube-system endpoints: - port: calico-metrics-port interval: 5s ``` To update Calico Cloud Prometheus' scrape interval to 10 seconds modify the manifest to this: ```yaml apiVersion: monitoring.coreos.com/v1alpha1 kind: ServiceMonitor metadata: name: calico-node-monitor namespace: tigera-prometheus labels: team: network-operators spec: selector: matchLabels: k8s-app: calico-node namespaceSelector: matchNames: - kube-system endpoints: - port: calico-metrics-port interval: 10s ``` ## Troubleshooting Config Updates[​](#troubleshooting-config-updates) Check config reloader logs to see if they detected any recent activity. - For prometheus run: ```bash kubectl -n tigera-prometheus logs prometheus- prometheus-config-reloader ``` - For alertmanager run: ```bash kubectl -n tigera-prometheus logs alertmanager- config-reloader ``` The config-reloaders watch each pods file-system for updated config from ConfigMap's or Secret's and will perform steps necessary for reloading the configuration. ### Configure Alertmanager Alertmanager is used by Calico Cloud to route alerts from Prometheus to the administrators. It handles routing, deduplicating, grouping, silencing and inhibition of alerts. More detailed information about Alertmanager is available in the [upstream documentation](https://prometheus.io/docs/alerting/latest/configuration). ### Updating the AlertManager config[​](#updating-the-alertmanager-config) - Save the current alertmanager secret, usually named `alertmanager-`. Our manifests will end up creating a secret called: `alertmanager-calico-node-alertmanager`. ```bash kubectl -n tigera-operator get secrets alertmanager-calico-node-alertmanager -o yaml > alertmanager-secret.yaml ``` - The current alertmanager.yaml file is encoded and stored inside the `alertmanager.yaml` key under the `data` field. You can decode it by copying the value of `alertmanager.yaml` and using the `base64` command. ```bash echo "" | base64 --decode > alertmanager-config.yaml ``` - Make necessary changes to `alertmanager-config.yaml`. Once this is done, you have to re-encode and save it to `alertmanager-secret.yaml`. You can do this by (in Linux): ```bash cat alertmanager-config.yaml | base64 -w 0 ``` - Paste the output of the running the command above back in `alertmanager-secret.yaml` replacing the value present in `alertmanager.yaml` field. Then apply this updated manifest. ```bash kubectl -n tigera-operator apply -f alertmanager-secret.yaml ``` Your changes should be applied in a few seconds by the config-reloader container inside the alertmanager pod launched by the prometheus-operator (usually named `alertmanager-`). For more advice on writing alertmanager configuration files, see the [alertmanager configuration](https://prometheus.io/docs/alerting/latest/configuration/) documentation. ### Configure Inhibition Rules[​](#configure-inhibition-rules) Alertmanager has a feature to suppress certain notifications according to defined rules. A typical use case for defining `inhibit` rules is to suppress notifications from a lower priority alert when one with a higher priority is firing. These inhibition rules are defined in the alertmanager configuration file. You can define one by adding this configuration snippet to your `alertmanager.yaml`. ```yaml [...] inhibit_rules: - source_match: severity: 'critical' target_match: severity: 'info' # Apply inhibition for alerts generated by the same alerting rule # and on the same node. equal: ['alertname', 'instance'] [...] ``` ### Configure Grouping of Alerts[​](#configure-grouping-of-alerts) Alertmanager also has a feature to group alerts based on labels and fine tune how often to resend an alert and so on. In the case of Denied Packet metrics, simply defining a Prometheus alerting rule would mean that you will get an page (if so defined in your alertmanager configuration) for every policy on every node for every Source IP. All these alerts can be combined into a single alert by configuring grouping. The Alertmanager configuration file that is provided with Calico Cloud by default, groups alerts on a per-node basis. Instead, if the goal is to group all alerts with the same name, edit (and apply) the alertmanager configuration file like so: ```yaml global: resolve_timeout: 5m route: group_by: ['alertname'] group_wait: 30s group_interval: 1m repeat_interval: 5m receiver: 'webhook' receivers: - name: 'webhook' webhook_configs: - url: 'http://calico-alertmanager-webhook:30501/' ``` More information, including descriptions of the various options can be found under the [route section](https://prometheus.io/docs/alerting/latest/configuration/#route) of the Alertmanager Configuration guide. ### Metrics ## [📄️ BGP metrics](https://docs.tigera.io/calico-cloud/operations/monitor/metrics/bgp-metrics) [Monitor BGP peering and route exchange in your cluster and get alerts by defining rules and thresholds.](https://docs.tigera.io/calico-cloud/operations/monitor/metrics/bgp-metrics) ## [📄️ Policy metrics](https://docs.tigera.io/calico-cloud/operations/monitor/metrics/policy-metrics) [Monitor the effects of policy in your cluster and received alerts by defining rules and thresholds.](https://docs.tigera.io/calico-cloud/operations/monitor/metrics/policy-metrics) ## [📄️ Fluentd metrics](https://docs.tigera.io/calico-cloud/operations/monitor/metrics/elasticsearch-and-fluentd-metrics) [Monitor Fluentd metrics and get alerts on log storage or collection issues.](https://docs.tigera.io/calico-cloud/operations/monitor/metrics/elasticsearch-and-fluentd-metrics) ### BGP metrics ## Big picture[​](#big-picture) Use Prometheus configured for Calico Cloud `calico-node` to monitor the health of BGP peers within your cluster. ## Value[​](#value) Using the open-source Prometheus monitoring and alerting toolkit, you can view time-series metrics from Calico Cloud components in the Prometheus or Grafana interfaces. Calico Cloud adds the ability to monitor high-level operations between BGP peers in your cluster. By defining a set of simple rules and thresholds, you can monitor peer-to-peer connection health between your nodes as well as the number of routes being exchanged and receive alerts when it exceeds configured thresholds. ## Concepts[​](#concepts) ```text +-------------------+ | Host | | +-------------------+ +------------+ +------------+ | | Host |------------->--| | | |--->-- | | +-------------------+ policy | Prometheus | | Prometheus | alert +-| | Host |----------->--| Server |-->--| Alert- |--->-- | | +-------------+ | metrics | | | manager | mechanisms +-| | BGP Metrics |-------------->--| | | |--->-- | | Server | | | | | | | +-------------+ | +------------+ +------------+ +-------------------+ ^ ^ | | Collect and store metrics. Web UI for accessing alert WebUI for accessing and states. querying metrics. Configure fan out Configure alerting rules. notifications to different alert receivers. ``` BGP metric reporting is accomplished using three key pieces: - BGP Metrics Server - Prometheus Server - Prometheus Alertmanager ### About Prometheus[​](#about-prometheus) The Prometheus scrapes various instrumented jobs (endpoints) to collect time series data for a given set of metrics. Time series data can then be queried and rules can be setup to monitor specific thresholds to trigger alerts. The data can also be visualized (such as using Grafana). Prometheus Server deployed as part of the Calico Cloud scrapes every configured `calico-node` target. Alerting rules querying BGP metrics can be configured in Prometheus and when triggered, fire alerts to the Prometheus Alertmanager. Prometheus Alertmanager (or simply Alertmanager), deployed as part of the Calico Cloud, receives alerts from Prometheus and forwards alerts to various alerting mechanisms such as *Pager Duty*, or *OpsGenie*. ### About Calico Cloud `calico-node`[​](#about-calico-cloud-calico-node) `calico-node` bundles together the components required for networking containers with Calico Cloud. The key components are: - Felix - BIRD - confd Its critical function means that it runs on every machine that provides endpoints. A binary running inside `calico-node` monitors the BIRD daemon for peering and routing activity and reports these statics to Prometheus. ## How to[​](#how-to) BGP metrics are generated within `calico-node` every 5 seconds using statistics pulled from the BIRD daemon. The metrics generated are: - `bgp_peers` - Total number of peers with a specific BGP connection status. - `bgp_routes_imported` - Current number of routes successfully imported into the routing table. - `bgp_route_updates_received` - Total number of route updates received over time (since startup). Calico Cloud will run BGP metrics for Prometheus by default. Metrics are directly available on each compute node at `http://:9900/metrics`. Refer to [Configuring Prometheus](https://docs.tigera.io/calico-cloud/operations/monitor/prometheus/) for information on how to create a new Alerting rule or updating the scraping interval for how often Prometheus collects the metrics. ### BGP peers metric[​](#bgp-peers-metric) The metric `bgp_peers` has the relevant labels `instance`, `status` and `ip_version`. Using this metric, you can identify how many peers have a specific BGP connection status with a given node instance and IP version. This metric will be available as a combination of `{instance, status, ip_version}`. Example queries: - Total number of peers currently with a BGP connection to the node instance “calico-node-1”, with status “Established”, for IP version “IPv4”. ```text bgp_peers{instance="calico-node-1", status="Established", ip_version="IPv4"} ``` - Total number of peers currently with a BGP connection to the node instance “calico-node-1”, with status “Down”, for IP version “IPv6”. ```text bgp_peers{instance="calico-node-1", status="Down", ip_version="IPv6"} ``` - Total number of peers currently with a BGP connection to any node instance, with a status that is not “Established”, for IP version “IPv4”. ```text bgp_peers{status!="Established", ip_version="IPv4"} ``` Valid BGP connection statuses are: "Idle", "Connect", "Active", "OpenSent", "OpenConfirm", "Established", "Close", "Down" and "Passive". ### BGP routes imported metric[​](#bgp-routes-imported-metric) The metric `bgp_routes_imported` has the relevant labels `instance` and `ip_version`. Using this metric, you can identify how many routes are being successfully imported into a given node instance's routing table at a specific point in time. This number can increase or decrease depending on how BGP rules process incoming routes. This metric will be available as a combination of `{instance, ip_version}`. Example queries: - Computes the per-second rate for the number of routes imported by a specific node instance “calico-node-1” looking up to 120 seconds back (using the two most recent data points). ```text irate(bgp_routes_imported{instance="calico-node-1",ip_version="IPv4"}[120s]) ``` - Computes the per-second rate for the number of routes imported across all node instances looking up to 120 seconds back (using the two most recent data points). ```text irate(bgp_routes_imported{ip_version="IPv4"}[120s]) ``` ### BGP route updates received metric[​](#bgp-route-updates-received-metric) The metric `bgp_route_updates_received` has the relevant labels `instance` and `ip_version`. Using this metric, you can identify the total number of BGP routes received by a given node over time. This number includes all routes that have been accepted & imported into the routing table, as well as any routes that were rejected as invalid, rejected by filters or rejected as already in the route table. This total number should only increase over time. This metric will be available as a combination of `{instance, ip_version}`. Example queries: - Computes the per-second rate for the number of routes received by a specific node instance “calico-node-1” looking up to 5 minutes back (using the two most recent data points). ```text irate(bgp_route_updates_received{instance="calico-node-1",ip_version="IPv4"}[5m]) ``` - Computes the per-second rate for the number of routes received across all node instances looking up to 5 minutes back (using the two most recent data points). ```text irate(bgp_route_updates_received{ip_version="IPv4"}[5m]) ``` ## Additional resources[​](#additional-resources) - [Secure Calico Cloud Prometheus endpoints](https://docs.tigera.io/calico-cloud/operations/comms/secure-metrics) - [Configuring Prometheus](https://docs.tigera.io/calico-cloud/operations/monitor/prometheus/) ### Policy metrics Calico Cloud adds the ability to monitor effects of policies configured in your cluster. By defining a set of simple rules and thresholds, you can monitor traffic metrics and receive alerts when it exceeds configured thresholds. ```text +------------+ | | | TSEE | | Manager | | | | | | | +------------+ ^ | | | +-----------------+ | | Host | | | +-----------------+ +------------+ +------------+ | | Host |------------->--| | | |--->-- | | +-----------------+ policy | Prometheus | | Prometheus | alert +-| | Host |----------->--| Server |-->--| Alert |--->-- | | +----------+ | metrics | | | Manager | mechanisms +-| | Felix |-------------->--| | | |--->-- | +----------+ | +------------+ +------------+ +-----------------+ ^ ^ | | Collect and store metrics. Web UI for accessing alert WebUI for accessing and states. querying metrics. Configure fan out Configure alerting rules. notifications to different alert receivers. ``` Policy inspection and reporting is accomplished using four key pieces: - A Calico Cloud specific Felix binary running inside `calico-node` container monitors the host for denied/allowed packets and collects metrics. - Prometheus Server(s) deployed as part of the Calico Cloud manifest scrapes every configured `calico-node` target. Alerting rules querying denied packet metrics are configured in Prometheus and when triggered, fire alerts to the Prometheus Alertmanager. - Prometheus Alertmanager (or simply Alertmanager), deployed as part of the Calico Cloud manifest, receives alerts from Prometheus and forwards alerts to various alerting mechanisms such as *Pager Duty*, or *OpsGenie*. - the Calico Cloud web console, also deployed as part of the Calico Cloud manifest, processes the metrics using pre-defined Prometheus queries and provides dashboards and associated workflows. Metrics will only be generated at a node when there are packets directed at an endpoint that are being actively profiled by a policy. Once generated they stay alive for 60 seconds. Once Prometheus scrapes a node and collects policy metrics, it will be available at Prometheus until the metric is considered *stale*, i.e., Prometheus has not seen any updates to this metric for some time. This time is configurable. Refer to [Configuring Prometheus configuration](https://docs.tigera.io/calico-cloud/operations/monitor/prometheus/) for more information. Because of metrics being expired, as just described, it is entirely possible for a GET on a metrics query URL to return no information. This is expected if there have not been any packets being processed by a policy on that node, in the last 60 seconds. Metrics generated by each Calico Cloud node are: - `calico_denied_packets` - Total number of packets denied by Calico Cloud policies. - `calico_denied_bytes` - Total number of bytes denied by Calico Cloud policies. - `cnx_policy_rule_packets` - Sum of allowed/denied packets over rules processed by Calico Cloud policies. - `cnx_policy_rule_bytes` - Sum of allowed/denied bytes over rules processed by Calico Cloud policies. - `cnx_policy_rule_connections` - Sum of connections over rules processed by Calico Cloud policies. The metrics `calico_denied_packets` and `calico_denied_bytes` have the labels `policy` and `srcIP`. Using these two metrics, one can identify the policy that denied packets as well as the source IP address of the packets that were denied by this policy. Using Prometheus terminology, `calico_denied_packets` is the metric name and `policy` and `srcIP` are labels. Each one of these metrics will be available as a combination of `{policy, srcIP}`. Example queries: - Total number of bytes, denied by Calico Cloud policies, originating from the IP address "10.245.13.133" by `k8s_ns.ns-0` profile. ```text calico_denied_bytes{policy="profile|k8s_ns.ns-0|0|deny", srcIP="10.245.13.133"} ``` - Total number of packets denied by Calico Cloud policies, originating from the IP address "10.245.13.149" by `k8s_ns.ns-0` profile. ```text calico_denied_packets{policy="profile|k8s_ns.ns-0|0|deny", srcIP="10.245.13.149"}} ``` The metrics `cnx_policy_rule_packets`, `cnx_policy_rule_bytes` and `cnx_policy_rule_connections` have the labels: `tier`, `policy`, `namespace`, `rule_index`, `action`, `traffic_direction`, `rule_direction`. Using these metrics, one can identify allow, and denied byte rate and packet rate, both inbound and outbound, indexed by both policy and rule. the Calico Cloud web console Dashboard makes heavy usage of these metrics. Staged policy names are prefixed with "staged:". Example queries: - Query counts for rules: Packet rates for specific rule by traffic\_direction ```text sum(irate(cnx_policy_rule_packets{namespace="namespace-2",policy="policy-0",rule_direction="ingress",rule_index="rule-5",tier="tier-0"}[30s])) without (instance) ``` - Query counts for rules: Packet rates for each rule in a policy by traffic\_direction ```text sum(irate(cnx_policy_rule_packets{namespace="namespace-2",policy="policy-0",tier="tier-0"}[30s])) without (instance) ``` - Query counts for a single policy by traffic\_direction and action ```text sum(irate(cnx_policy_rule_packets{namespace="namespace-2",policy="policy-0",tier="tier-0"}[30s])) without (instance,rule_index,rule_direction) ``` - Query counts for all policies across all tiers by traffic\_direction and action ```text sum(irate(cnx_policy_rule_packets[30s])) without (instance,rule_index,rule_direction) ``` See the [Felix configuration reference](https://docs.tigera.io/calico-cloud/reference/component-resources/node/felix/configuration#calico-cloud-specific-configuration) for the settings that control the reporting of these metrics. Calico Cloud manifests normally set `PrometheusReporterEnabled=true` and `PrometheusReporterPort=9081`, so these metrics are available on each compute node at `http://:9081/metrics`. ### Fluentd metrics ## Big picture[​](#big-picture) Use the Prometheus monitoring and alerting tool for Fluentd metrics to ensure continuous network visibility. ## Value[​](#value) Platform engineering teams rely on logs for visibility into their networks. If collecting or storing logs are disrupted, this can impact network visibility. Prometheus can monitor log collection and storage metrics so platform engineering teams are alerted about problems before they occur. ## Concepts[​](#concepts) | Component | Description | | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Prometheus | Monitoring tool that scrapes metrics from instrumented jobs and displays time series data in a visualizer (such as Grafana). For Calico Cloud, the “jobs” that Prometheus can harvest metrics from the Fluentd component. | | Fluentd | Sends Calico Cloud logs to Elasticsearch for storage. | ## How to[​](#how-to) ### Create Prometheus alerts for Fluentd[​](#create-prometheus-alerts-for-fluentd) The following example creates a Prometheus rule to monitor some important Fluentd metrics, and alert when they have crossed certain thresholds: ```yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: tigera-prometheus-log-collection-monitoring namespace: tigera-prometheus labels: role: tigera-prometheus-rules prometheus: calico-node-prometheus spec: groups: - name: tigera-log-collection.rules rules: - alert: FluentdPodConsistentlyLowBufferSpace expr: avg_over_time(fluentd_output_status_buffer_available_space_ratio[5m]) < 75 labels: severity: Warning annotations: summary: "Fluentd pod {{$labels.pod}}'s buffer space is consistently below 75 percent capacity." description: "Fluentd pod {{$labels.pod}} has very low buffer space. There may be connection issues between Elasticsearch and Fluentd or there are too many logs to write out, check the logs for the Fluentd pod." ``` #### The alerts created in the example are described as follows:[​](#the-alerts-created-in-the-example-are-described-as-follows) | Alert | Severity | Requires | Issue/reason | | ---------------------------------------- | --------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **FluentdPodConsistentlyLowBufferSpace** | Non-critical, warning | Immediate investigation to ensure logs are being gathered correctly. | A Fluentd pod’s available buffer size has averaged less than 75% over the last 5 minutes. This could mean Fluentd is having trouble communicating with the Elasticsearch cluster, the Elasticsearch cluster is down, or there are simply too many logs to process. | ### eBPF data plane mode ## [📄️ eBPF use cases](https://docs.tigera.io/calico-cloud/operations/ebpf/use-cases-ebpf) [Learn when to use eBPF, and when not to.](https://docs.tigera.io/calico-cloud/operations/ebpf/use-cases-ebpf) ## [📄️ Enable eBPF on an existing cluster](https://docs.tigera.io/calico-cloud/operations/ebpf/enabling-ebpf) [Steps to enable the eBPF data plane on an existing cluster.](https://docs.tigera.io/calico-cloud/operations/ebpf/enabling-ebpf) ## [📄️ Troubleshoot eBPF mode](https://docs.tigera.io/calico-cloud/operations/ebpf/troubleshoot-ebpf) [How to troubleshoot when running in eBPF mode.](https://docs.tigera.io/calico-cloud/operations/ebpf/troubleshoot-ebpf) ### eBPF use cases ## Big picture[​](#big-picture) Learn when to use eBPF (and when not to). ## What is eBPF?[​](#what-is-ebpf) eBPF is a feature available in Linux kernels that allows you to run a virtual machine inside the kernel. This virtual machine allows you to safely load programs into the kernel, to customize its operation. Why is this important? In the past, making changes to the kernel was difficult: there were APIs you could call to get data, but you couldn’t influence what was inside the kernel or execute code. Instead, you had to submit a patch to the Linux community and wait for it to be approved. With eBPF, you can load a program into the kernel and instruct the kernel to execute your program if, for example, a certain packet is seen or another event occurs. With eBPF, the kernel and its behavior become highly customizable, instead of being fixed. This can be extremely beneficial, when used under the right circumstances. ## Calico Cloud and eBPF[​](#calico-cloud-and-ebpf) Calico Cloud offers an eBPF data plane as an alternative to our standard Linux data plane (which is iptables based). While the standard data plane focuses on compatibility by working together with kube-proxy and your own iptables rules, the eBPF data plane focuses on performance, latency, and improving user experience with features that aren’t possible with the standard data plane. But Calico Cloud doesn’t only support standard Linux and eBPF; it currently supports a total of three data planes, including Windows HNS, and has plans to add support for even more data planes in the near future. Calico Cloud enables you, the user, to decide what works best for what you want to do. If you enable eBPF within Calico Cloud but have existing iptables flows, we won’t touch them. Because maybe you want to use connect-time load balancing, but leave iptables as is. With Calico Cloud, it’s not an all-or-nothing deal—we allow you to easily load and unload our eBPF data plane to suit your needs, which means you can quickly try it out before making a decision. Calico Cloud offers you the ability to leverage eBPF as needed, as an additional control to build your Kubernetes cluster security. ## Use cases[​](#use-cases) There are several use cases for eBPF, including traffic control, creating network policy, and connect-time load balancing. ### Traffic control[​](#traffic-control) Without eBPF, packets use the standard Linux networking path on their way to a final destination. If a packet shows up at point A, and you know that the packet needs to go to point B, you can optimize the network path in the Linux kernel by sending it straight to point B. With eBPF, you can leverage additional context to make these changes in the kernel so that packets bypass complex routing and simply arrive at their final destination. This is especially relevant in a Kubernetes container environment, where you have numerous networks. (In addition to the host network stack, each container has its own mini network stack.) When traffic comes in, it is usually routed to a container stack and must travel a complex path as it makes its way there from the host stack. This routing can be bypassed using eBPF. ### Creating network policy[​](#creating-network-policy) When creating network policy, there are two instances where eBPF can be used: - **eXpress Data Path (XDP)** – As a raw packet buffer enters the system, eBPF gives you an efficient way to examine that buffer and make quick decisions about what to do with it. - **Network policy** – eBPF allows you to efficiently examine a packet and apply network policy, both for pods and hosts. ### Connect-time load balancing[​](#connect-time-load-balancing) When load balancing service connections in Kubernetes, a port needs to talk to a service and therefore network address translation (NAT) must occur. A packet is sent to a virtual IP, and that virtual IP translates it to the destination IP of the pod backing the service; the pod then responds to the virtual IP and the return packet is translated back to the source. With eBPF, you can avoid this packet translation by using an eBPF program that you’ve loaded into the kernel and load balancing at the source of the connection. All NAT overhead from service connections is removed because destination network address translation (DNAT) does not need to take place on the packet processing path. ## The price of performance[​](#the-price-of-performance) So is eBPF more efficient than standard Linux iptables? The short answer: it depends. If you were to micro-benchmark how iptables works when applying network policies with a large number of IP addresses (i.e. ipsets), iptables in many cases is better than eBPF. But if you want to do something in the Linux kernel where you need to alter the packet flow in the kernel, eBPF would be the better choice. Standard Linux iptables is a complex system and certainly has its limitations, but at the same time it provides options to manipulate traffic; if you know how to program iptables rules, you can achieve a lot. eBPF allows you to load your own programs into the kernel to influence behavior that can be customized to your needs, so it is more flexible than iptables as it is not limited to one set of rules. Something else to consider is that, while eBPF allows you to run a program, add logic, redirect flows, and bypass processing—which is a definite win—it’s a virtual machine and as such must be translated to bytecode. By comparison, the Linux kernel’s iptables is already compiled to code. As you can see, comparing eBPF to iptables is not a straight apples-to-apples comparison. What we need to assess is performance, and the two key factors to look at here are latency (speed) and expense. If eBPF is very fast but takes up 80% of your resources, then it’s like a Lamborghini—an expensive, fast car. And if that works for you, great (maybe you really like expensive, fast cars). Just keep in mind that more CPU usage means more money spent with your cloud providers. So while a Lamborghini might be faster than a lot of other cars, it might not be the best use of money if you need to comply with speed limits on your daily commute. ## When to use eBPF (and when not to)[​](#when-to-use-ebpf-and-when-not-to) With eBPF, you get performance—but it comes at a cost. You need to find a balance between the two by figuring out the price of performance, and deciding if it’s acceptable to you from an eBPF perspective. Let’s look at some specific cases where it would make sense to use eBPF, and some where it would not. ### When not to use eBPF[​](#when-not-to-use-ebpf) ### ✘ Packet-by-packet processing[​](#-packet-by-packet-processing) Using eBPF to perform CPU intensive or packet-by-packet processing, such as decryption and re-encryption for encrypted flows, would not be efficient because you would need to build a structure and do a lookup for every packet, which is expensive. ### When to use eBPF[​](#when-to-use-ebpf) ### ✔ XDP[​](#-xdp) eBPF provides an efficient way to examine raw packet buffers as they enter the system, allowing you to make quick decisions about what to do with them. ### ✔ Connect-time load balancing[​](#-connect-time-load-balancing) With eBPF, you can load balance at the source using a program you’ve loaded into the kernel, instead of using a virtual IP. Since DNAT does not need to take place on the packet processing path, all NAT overhead from service connections is removed. ### ✔ Building a service mesh control plane[​](#-building-a-service-mesh-control-plane) Service mesh relies on proxies like Envoy. A lot of thought has gone into designing this process over the years. The main reason for doing it this way is that, in many cases, it is not viable to do inline processing for application protocols like HTTP at the high speeds seen inside a cluster. Therefore, you should think of using eBPF to route traffic to a proxy like Envoy in an efficient way, rather than using it to replace the proxy itself. However, you do need to turn off connect-time load balancing (CTLB) so sidecars can see the service addresses. Given you are already taking a performance hit by the extra hop to the sidecar, not using CTLB performance optimization to avoid NAT overhead is likely not a big deal. ## Summary[​](#summary) Is eBPF a replacement for iptables? Not exactly. It’s hard to imagine everything working as efficiently with eBPF as it does with iptables. For now, the two co-exist and it’s up to the user to weigh the price-performance tradeoff and decide which feature to use when, given their specific needs. We believe the right solution is to leverage eBPF, along with existing mechanisms in the Linux kernel, to achieve your desired outcome. That’s why Calico Cloud offers support for multiple data planes, including standard Linux, Windows HNS, and Linux eBPF. Since we have established that both eBPF and iptables are useful, the only logical thing to do in our opinion is to support both. Calico Cloud gives you the choice so you can choose the best tool for the job. ## Additional resources[​](#additional-resources) To learn more and see performance metrics from our test environment, see the blog, [Introducing the eBPF data plane](https://www.projectcalico.org/introducing-the-calico-ebpf-dataplane/). ### Enable eBPF on an existing cluster ## Big picture[​](#big-picture) Enable the eBPF data plane on an existing cluster. ## Value[​](#value) The eBPF data plane mode has several advantages over standard Linux networking pipeline mode: - It scales to higher throughput. - It uses less CPU per GBit. - It has native support for Kubernetes services (without needing kube-proxy) that: - Reduces first packet latency for packets to services. - Preserves external client source IP addresses all the way to the pod. - Supports DSR (Direct Server Return) for more efficient service routing. - Uses less CPU than kube-proxy to keep the data plane in sync. To learn more and see performance metrics from our test environment, see the blog, [Introducing the Calico eBPF data plane](https://www.projectcalico.org/introducing-the-calico-ebpf-dataplane/). ## Concepts[​](#concepts) ### eBPF[​](#ebpf) eBPF (or "extended Berkeley Packet Filter"), is a technology that allows safe mini programs to be attached to various low-level hooks in the Linux kernel. eBPF has a wide variety of uses, including networking, security, and tracing. You’ll see a lot of non-networking projects leveraging eBPF, but for Calico Cloud our focus is on networking, and in particular, pushing the networking capabilities of the latest Linux kernels to the limit. ## Before you begin[​](#before-you-begin) **Supported architecture and versions** - x86-64 - arm64 (little-endian) - Linux distribution/kernel: - Ubuntu 20.04 or above. - Red Hat v8.2 with Linux kernel v4.18.0-193 or above (Red Hat have backported the required features to that build). - Another supported distribution with Linux kernel v5.3 or above. - Immutable operating systems (e.g., Talos Linux): Ensure the `CgroupV2Path` in the `FelixConfiguration` CRD is set to a writable path. #### Kernel version requirements for eBPF features[​](#kernel-version-requirements-for-ebpf-features) Some eBPF features require a higher kernel version than the base eBPF data plane. | Feature | Minimum kernel version | Details | | -------------------------------------------------------------------------------------------------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Base eBPF data plane | v5.3 (RHEL: v4.18.0-193) | v5.8+ with CO-RE recommended for better performance | | [XDP acceleration](https://docs.tigera.io/calico-cloud/network-policy/extreme-traffic/defend-dos-attack) | v4.16 | Used for DoS mitigation; when `bpfEnabled` is `true`, policy is always accelerated using best available BPF technology | | Log rules in eBPF mode | v5.16 | Logs sent to trace pipe via `bpftool prog tracelog` | | [QoS bandwidth controls](https://docs.tigera.io/calico-cloud/networking/configuring/qos-controls) | v6.6 | Requires `tcx` support. Established connection limits not supported with eBPF | | [DNS policy inline mode](https://docs.tigera.io/calico-cloud/network-policy/domain-based-policy) | v5.17 (RHEL: v5.14) | `BPFDNSPolicyMode: Inline` parses DNS responses in eBPF before they reach the application. Only wildcard prefixes (`*.x.y.z`) supported. Falls back to `NoDelay` on older kernels | > **WARNING:** While v5.3 is the minimum kernel version required for the eBPF data plane, we strongly recommend using kernel v5.8 or above, which adds support for [CO-RE (Compile Once - Run Everywhere)](https://docs.ebpf.io/concepts/core/). CO-RE significantly improves compatibility and performance across kernel versions. For access to all eBPF features, we recommend kernel v6.6 or above. - An underlying network fabric that allows VXLAN traffic between hosts. In eBPF mode, VXLAN is used to forward Kubernetes NodePort traffic. **Unsupported platforms** - GKE - MKE - TKG > **SECONDARY:** eBPF supports AKS with Calico CNI and Calico Cloud network policy. However, with AKS with Azure CNI and Calico Cloud network policy, kube-proxy cannot be disabled so the performance benefits of eBPF are lost. However, there are other reasons to use eBPF other than performance gains, as described in [eBPF use cases](https://docs.tigera.io/calico-cloud/operations/ebpf/use-cases-ebpf). **Unsupported features** - Clusters with some eBPF nodes and some standard data plane and/or Windows nodes - IPv6 - Host endpoint `doNotTrack` policy (other policy types are supported) - Floating IPs - SCTP (either for policy or services) - Tagged VLAN devices - L7 logs - Application layer policies - Web application firewall (WAF) **Recommendations for performance** For best pod-to-pod performance, we recommend using an underlying network that doesn't require Calico Cloud to use an overlay. For example: - A cluster within a single AWS subnet - A cluster using a compatible cloud provider's CNI (such as the AWS VPC CNI plugin) - An on-prem cluster with BGP peering configured If you must use an overlay, we recommend that you use VXLAN, not IPIP. VXLAN has better performance than IPIP in eBPF mode due to various kernel optimizations. ## How to[​](#how-to) - [Verify that your cluster is ready for eBPF mode](#verify-that-your-cluster-is-ready-for-ebpf-mode) - [Configure Calico Cloud to talk directly to the API server](#configure-calico-cloud-to-talk-directly-to-the-api-server) - [Configure kube-proxy](#configure-kube-proxy) - [Enable eBPF mode](#enable-ebpf-mode) - [Try out DSR mode](#try-out-direct-server-return-mode) - [Reversing the process](#reversing-the-process) ### Verify that your cluster is ready for eBPF mode[​](#verify-that-your-cluster-is-ready-for-ebpf-mode) This section explains how to make sure your cluster is suitable for eBPF mode. To check that the kernel on a node is suitable, you can run ```bash uname -rv ``` The output should look like this: ```text 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 ``` In this case the kernel version is v5.4, which is suitable. On Red Hat-derived distributions, you may see something like this: ```text 4.18.0-193.el8.x86_64 (mockbuild@x86-vm-08.build.eng.bos.redhat.com) ``` Since the Red Hat kernel is v4.18 with at least build number 193, this kernel is suitable. ### Configure Calico Cloud to talk directly to the API server[​](#configure-calico-cloud-to-talk-directly-to-the-api-server) In eBPF mode, Calico Cloud implements Kubernetes service networking directly (rather than relying on `kube-proxy`). Of course, this makes it highly desirable to disable `kube-proxy` when running in eBPF mode to save resources and avoid confusion over which component is handling services. To be able to disable `kube-proxy`, Calico Cloud needs to communicate to the API server *directly* rather than going through `kube-proxy`. To make *that* possible, we need to find a persistent, static way to reach the API server. The best way to do that varies by Kubernetes distribution: - If you created a cluster manually (for example by using `kubeadm`) then the right address to use depends on whether you opted for a high-availability cluster with multiple API servers or a simple one-node API server. - If you opted to set up a high availability cluster then you should use the address of the load balancer that you used in front of your API servers. As noted in the Kubernetes documentation, a load balancer is required for a HA set-up but the precise type of load balancer is not specified. - If you opted for a single control plane node then you can use the address of the control plane node itself. However, it's important that you use a *stable* address for that node such as a dedicated DNS record, or a static IP address. If you use a dynamic IP address (such as an EC2 private IP) then the address may change when the node is restarted causing Calico Cloud to lose connectivity to the API server. - `kops` typically sets up a load balancer of some sort in front of the API server. You should use the FQDN and port of the API load balancer, for example `api.internal.` as the `KUBERNETES_SERVICE_HOST` below and 443 as the `KUBERNETES_SERVICE_PORT`. - OpenShift requires various DNS records to be created for the cluster; one of these is exactly what we need: `api-int..` should point to the API server or to the load balancer in front of the API server. Use that (filling in the `` and `` as appropriate for your cluster) for the `KUBERNETES_SERVICE_HOST` below. Openshift uses 6443 for the `KUBERNETES_SERVICE_PORT`. - For AKS and EKS clusters you should use the FQDN of the API server's load balancer. This can be found with ```text kubectl cluster-info ``` which gives output like the following: ```text Kubernetes master is running at https://60F939227672BC3D5A1B3EC9744B2B21.gr7.us-west-2.eks.amazonaws.com ... ``` In this example, you would use `60F939227672BC3D5A1B3EC9744B2B21.gr7.us-west-2.eks.amazonaws.com` for `KUBERNETES_SERVICE_HOST` and `443` for `KUBERNETES_SERVICE_PORT` when creating the config map. Once you've found the correct address for your API server, create the following config map in the `tigera-operator` namespace using the host and port that you found above: ```yaml kind: ConfigMap apiVersion: v1 metadata: name: kubernetes-services-endpoint namespace: tigera-operator data: KUBERNETES_SERVICE_HOST: '' KUBERNETES_SERVICE_PORT: '' ``` The operator will pick up the change to the config map automatically and do a rolling update of Calico Cloud to pass on the change. Confirm that pods restart and then reach the `Running` state with the following command: ```text watch kubectl get pods -n calico-system ``` If you do not see the pods restart then it's possible that the `ConfigMap` wasn't picked up (sometimes Kubernetes is slow to propagate `ConfigMap`s (see Kubernetes [issue #30189](https://github.com/kubernetes/kubernetes/issues/30189))). You can try restarting the operator. ### Configure kube-proxy[​](#configure-kube-proxy) In eBPF mode Calico Cloud replaces `kube-proxy` so it wastes resources (and reduces performance) to run both.\ This section explains how to disable `kube-proxy` in some common environments. #### Clusters that run `kube-proxy` with a `DaemonSet` (such as `kubeadm`)[​](#clusters-that-run-kube-proxy-with-a-daemonset-such-as-kubeadm) For a cluster that runs `kube-proxy` in a `DaemonSet` (such as a `kubeadm`-created cluster), you can disable `kube-proxy` reversibly by adding a node selector to `kube-proxy`'s `DaemonSet` that matches no nodes, for example: ```text kubectl patch ds -n kube-system kube-proxy -p '{"spec":{"template":{"spec":{"nodeSelector":{"non-calico": "true"}}}}}' ``` Then, should you want to start `kube-proxy` again, you can simply remove the node selector. > **SECONDARY:** This approach is not suitable for AKS with Azure CNI since that platform makes use of the Kubernetes add-on manager. the change will be reverted by the system. For AKS, you should follow [Avoiding conflicts with kube-proxy](#avoiding-conflicts-with-kube-proxy) below. #### OpenShift[​](#openshift) If you are running OpenShift, you can disable `kube-proxy` as follows: ```text kubectl patch networks.operator.openshift.io cluster --type merge -p '{"spec":{"deployKubeProxy": false}}' ``` To re-enable it: ```text kubectl patch networks.operator.openshift.io cluster --type merge -p '{"spec":{"deployKubeProxy": true}}' ``` ### Avoiding conflicts with kube-proxy[​](#avoiding-conflicts-with-kube-proxy) If you cannot disable `kube-proxy` (for example, because it is managed by your Kubernetes distribution), then you *must* change Felix configuration parameter `BPFKubeProxyIptablesCleanupEnabled` to `false`. This can be done with `kubectl` as follows: ```text kubectl patch felixconfiguration default --patch='{"spec": {"bpfKubeProxyIptablesCleanupEnabled": false}}' ``` If both `kube-proxy` and `BPFKubeProxyIptablesCleanupEnabled` is enabled then `kube-proxy` will write its iptables rules and Felix will try to clean them up resulting in iptables flapping between the two. ### Enable eBPF mode[​](#enable-ebpf-mode) To enable eBPF mode, change the `spec.calicoNetwork.linuxDataplane` parameter in the operator's `Installation` resource to `"BPF"`. ```bash kubectl patch installation.operator.tigera.io default --type merge -p '{"spec":{"calicoNetwork":{"linuxDataplane":"BPF"}}}' ``` When enabling eBPF mode, preexisting connections continue to use the non-BPF datapath; such connections should not be disrupted, but they do not benefit from eBPF mode’s advantages. > **SECONDARY:** The operator rolls out the change with a rolling update (non-disruptive) and then swiftly transitions all nodes to eBPF mode. However, it's inevitable that some nodes will enter eBPF mode before others. This can disrupt the flow of traffic through node ports. ### Try out direct server return mode[​](#try-out-direct-server-return-mode) Direct server return (DSR) mode skips a hop through the network for traffic to services (such as node ports) from outside the cluster. This reduces latency and CPU overhead but it requires the underlying network to allow nodes to send traffic with each other's IPs. In AWS, this requires all your nodes to be in the same subnet and for the source/dest check to be disabled. In GCP, the source/dest check should also be disabled, which can be done by enabling IP forwarding. DSR mode is disabled by default; to enable it, set the `BPFExternalServiceMode` Felix configuration parameter to `"DSR"`. This can be done with `kubectl`: ```text kubectl patch felixconfiguration default --patch='{"spec": {"bpfExternalServiceMode": "DSR"}}' ``` To switch back to tunneled mode, set the configuration parameter to `"Tunnel"`: ```text kubectl patch felixconfiguration default --patch='{"spec": {"bpfExternalServiceMode": "Tunnel"}}' ``` Switching external traffic mode can disrupt in-progress connections. ### Reversing the process[​](#reversing-the-process) To revert to standard Linux networking: 1. Reverse the changes to the operator's `Installation`: ```bash kubectl patch installation.operator.tigera.io default --type merge -p '{"spec":{"calicoNetwork":{"linuxDataplane":"Iptables"}}}' ``` 2. If you disabled `kube-proxy`, re-enable it (for example, by removing the node selector added above). ```text kubectl patch ds -n kube-system kube-proxy --type merge -p '{"spec":{"template":{"spec":{"nodeSelector":{"non-calico": null}}}}}' ``` 3. Since disabling eBPF mode is disruptive to existing connections, monitor existing workloads to make sure they re-establish any connections that were disrupted by the switch. ### Troubleshoot eBPF mode This document gives general troubleshooting guidance for the eBPF data plane. To understand basic concepts, we recommend the following video by Tigera Engineers: [Opening the Black Box: Understanding and troubleshooting Calico's eBPF Data Plane](https://www.youtube.com/watch?v=Mh43sNBu208). ## Troubleshoot access to services[​](#troubleshoot-access-to-services) - Verify that eBPF mode is correctly enabled Examine the log for a `calico-node` container; in the extremely rare case when eBPF mode is not supported it will log an `ERROR` log that says ```bash BPF data plane mode enabled but not supported by the kernel. Disabling BPF mode. ``` If BPF mode is correctly enabled, you should see an `INFO` log that says ```bash BPF enabled, starting BPF endpoint manager and map manager. ``` - In eBPF mode, forwarding external client access to services (typically NodePorts) from node to node is implemented using VXLAN encapsulation. If NodePorts time out when the backing pod is on another node, check your underlying network fabric allows VXLAN traffic between the nodes. VXLAN is a UDP protocol; by default it uses port 4789. Note that this VXLAN traffic is separate from any overlay network that you may be using for pod-to-pod traffic. - In DSR mode, Calico Cloud requires that the underlying network fabric allows one node to respond on behalf of another. - In AWS, to allow this, the Source/Dest check must be disabled on the node's NIC. However, note that DSR only works within AWS; it is not compatible with external traffic through a load balancer. This is because the load balancer is expecting the traffic to return from the same host. - In GCP, the "Allow forwarding" option must be enabled. As with AWS, traffic through a load balancer does not work correctly with DSR because the load balancer is not consulted on the return path from the backing node. # The `calico-node -bpf` tool To inspect Calico Cloud's internal data structures, you can use the `calico-node -bpf` tool. The tool is embedded in the cnx-node container image and displays information about the eBPF data plane from within a `calico-node` pod only. Use `kubectl get pod -o wide -n calico-system` to find the name of a `calico-node` pod and use the name in the following commands instead of ``. To run the tool, use: ```bash kubectl exec -n calico-system -- calico-node -bpf ``` For example, to show the tool's help: ```bash kubectl exec -n calico-system -- calico-node -bpf help Available Commands: arp Manipulates arp cleanup Removes all calico-bpf programs and maps completion Generate the autocompletion script for the specified shell connect-time Manipulates connect-time load balancing programs conntrack Manipulates connection tracking counters Show and reset counters help Help about any command ifstate Manipulates ifstate ipsets Manipulates ipsets nat Manipulates network address translation (nat) policy Dump policy attached to interface profiling Show and reset profiling data routes Manipulates routes version Prints the version and exits (Since the tool is embedded in the main `calico-node` binary the `--help` option is not available, but running `calico-node -bpf help` does work.) ``` For example, to dump the BPF conntrack table, use: ```text kubectl exec -n calico-system -- calico-node -bpf conntrack dump ... ``` ## Debug access to services[​](#debug-access-to-services) Inspect the BPF NAT table to verify that the service is correctly programmed. To dump the BPF NAT table: ```text kubectl exec -n calico-system -- calico-node -bpf nat dump 10.96.0.10 port 53 proto 6 id 4 count 2 local 0 4:0 192.168.129.66:53 4:1 192.168.129.68:53 10.96.0.10 port 53 proto 17 id 6 count 2 local 0 6:0 192.168.129.66:53 6:1 192.168.129.68:53 10.96.0.10 port 9153 proto 6 id 5 count 2 local 0 5:0 192.168.129.66:9153 5:1 192.168.129.68:9153 10.105.77.92 port 5473 proto 6 id 0 count 2 local 0 0:0 10.128.1.192:5473 0:1 10.128.1.195:5473 10.105.187.231 port 8081 proto 6 id 2 count 1 local 0 2:0 192.168.105.131:8081 10.109.136.88 port 7443 proto 6 id 1 count 1 local 0 1:0 192.168.129.72:7443 10.109.139.39 port 443 proto 6 id 7 count 2 local 0 7:0 192.168.129.67:5443 7:1 192.168.129.69:5443 10.96.0.1 port 443 proto 6 id 3 count 1 local 0 3:0 10.128.0.255:6443 ``` Inspect the BPF conntrack table to verify that connections are being tracked. To dump the BPF conntrack table: ```text kubectl exec -n calico-system -- calico-node -bpf conntrack dump TCP 192.168.58.7:49178 -> 10.111.57.87:80 -> 192.168.105.136:80 Active ago 4.486606371s CLOSED <--- example of connection to service with per-packet NAT TCP 10.128.1.194:41513 -> 10.128.1.192:179 Active ago 26.442759238s ESTABLISHED <--- example of connection without NAT or with connect-time NAT TCP 192.168.58.7:42818 -> 10.111.57.87:80 -> 192.168.105.136:80 Active ago 1m15.987585857s CLOSED TCP 10.128.1.192:58208 -> 10.109.136.88:7443 -> 192.168.58.5:7443 Active ago 4.935017508s ESTABLISHED UDP 162.142.125.240:30603 -> 10.128.1.192:18989 Active ago 1m0.816678617s UDP 127.0.0.1:48611 -> 127.0.0.53:53 Active ago 17.789851961s ``` Note that traffic originating within the cluster uses connect-time load balancing. By default, the connect-time load balancing is only enabled for TCP traffic. When connect-time load balancing is used, the conntrack table will not show the NAT resolution as that happens when the application calls `connect()`. ## Check if Calico Cloud is dropping packets[​](#check-if-calico-cloud-is-dropping-packets) If you suspect that Calico Cloud is dropping packets, you can use the `calico-node -bpf` tool to check the BPF counters. Since the eBPF data plane is split into programs that are attached to interfaces, you must check the counters on the relevant interface. You can either dump counters for all interfaces or use `--iface=` to dump counters for a specific interface. Increasing counter `Dropped by policy` indicates that Calico Cloud is dropping packets due to policy and you should check your policy configuration. ```text kubectl exec -n calico-system -- calico-node -bpf counters dump --iface=eth0 +----------+--------------------------------+---------+--------+-----+ | CATEGORY | TYPE | INGRESS | EGRESS | XDP | +----------+--------------------------------+---------+--------+-----+ | Accepted | by another program | 0 | 0 | N/A | | | by failsafe | 0 | 0 | N/A | | | by policy | 0 | 4 | N/A | | Dropped | NAT source collision | 0 | 0 | N/A | | | resolution failed | | | | | | QoS control limit | 0 | 0 | N/A | | | by policy | 0 | 11 | N/A | | | failed decapsulation | 0 | 0 | N/A | | | failed encapsulation | 0 | 0 | N/A | | | failed to create conntrack | 0 | 0 | N/A | | | fragment of yet incomplete | 0 | 0 | N/A | | | packet | | | | | | fragment out of order within | 0 | 0 | N/A | | | host | | | | | | fragments not supported | 0 | 0 | N/A | | | incorrect checksum | 0 | 0 | N/A | | | malformed IP packets | 0 | 0 | N/A | | | packets hitting blackhole | 0 | 0 | N/A | | | route | | | | | | packets with unknown route | 0 | 0 | N/A | | | packets with unknown source | 0 | 0 | N/A | | | packets with unsupported IP | 0 | 0 | N/A | | | options | | | | | | too short packets | 0 | 0 | N/A | | Other | packets hitting NAT source | 0 | 0 | N/A | | | collision | | | | | Redirect | neigh | 0 | 0 | N/A | | | peer | 0 | 0 | N/A | | | plain | 20 | 0 | N/A | | Total | packets | 34 | 22 | N/A | +----------+--------------------------------+---------+--------+-----+ ``` ## eBPF program debug logs[​](#ebpf-program-debug-logs) Sometimes it is necessary to examine the logs that are emitted by the eBPF programs themselves. Although the logs can be very verbose (because the programs will log every packet), they can be invaluable to diagnose eBPF program issues. To enable the log, set the `bpfLogLevel` Felix configuration setting to `Debug`. > **WARNING:** Enabling logs in this way has a significant impact on eBPF program performance. To reduce the performance impact in production clusters, you can target logging to specific traffic and/or specific interfaces using the [bpfLogFilters](https://docs.tigera.io/calico-cloud/reference/component-resources/node/felix/configuration#BPFLogFilters) Felix configuration setting. Filters are pcap expressions. > Note that the filters are applied to the original packet, before any NAT or encapsulation. Therefore, to log a packet that is being sent to a service making its way via different devices, you must filter on the service IP and port and also the backend pod IP and port. The logs are emitted to the kernel trace buffer, and they can be examined using the following command: ```text kubectl exec -n calico-system -- bpftool prog tracelog ``` Logs have the following format: ```text <...>-84582 [000] .Ns1 6851.690474: 0: ens192---E: Final result=ALLOW (-1). Program execution time: 7366ns ``` The parts of the log are explained below: - `<...>-84582` gives an indication about what program (or kernel process) was handling the packet. For packets that are being sent, this is usually the name and PID of the program that is actually sending the packet. For packets that are received, it is typically a kernel process, or an unrelated program that happens to trigger the processing. - `6851.690474` is the log timestamp. - `ens192---E` is the Calico Cloud log tag. For programs attached to interfaces, the first part contains the first few characters of the interface name. The suffix is either `-I` or `-E` indicating "Ingress" or "Egress". "Ingress" and "Egress" have the same meaning as for policy: - A workload ingress program is executed on the path from the host network namespace to the workload. - A workload egress program is executed on the workload to host path. - A host endpoint ingress program is executed on the path from external node to the host. - A host endpoint egress program is executed on the path from host to external host. - you may also see `ens192---X` which indicates an XDP program. Calico Cloud uses XDP programs to implement `doNotTrack` policies on host devices only. - `Final result=ALLOW (-1). Program execution time: 7366ns` is the message. In this case, logging the final result of the program. Note that the timestamp is massively distorted by the time spent logging. ## Debugging policy issues[​](#debugging-policy-issues) If you suspect that Calico Cloud is dropping packets due to policy, you can use the `calico-node -bpf` tool to dump the policy that is attached to a specific interface. ```text kubectl exec -n calico-system -- calico-node -bpf policy dump [--asm] ``` Where: - `` is the name of the interface, for example `eth0` or `caliXXXXXX`. - `` represents location of the policy, either `ingress`, `egress`, `xdp` or `all`. Dump of an ingress policy. Note that ingress policy for a pod is attached to the `tc/tcx` egress hook of the host-side of the caliX veth pair, while ingress policy for host endpoints is attached to the `tc/tcx` ingress hook of the host interface. Similarly for egress policy. ```text IfaceName: calic31b4f7fc58 Hook: tc egress Error: Policy Info: // Start of tier default // Start of policy default/knp.default.allow-nginx-from-ubuntu // Start of rule action:"allow" protocol:{name:"tcp"} dst_ports:{first:80 last:80} src_ip_set_ids:"s:nzE8vwTu69FSscx2FDKjb20D9dZxEyVxsWFqwA" original_src_selector:"projectcalico.org/orchestrator == 'k8s' && app == 'ubuntu-client'" rule_id:"29vMYcPWr7reSxxN" // IPSets src_ip_set_ids:<0x303904ae5eae5418> // count = 9 // End of rule 29vMYcPWr7reSxxN // End of policy default/knp.default.allow-nginx-from-ubuntu // End of tier default: deny // Start of rule action:"allow" rule_id:"aBMQCbsUMESPKGRp" // count = 0 // End of rule aBMQCbsUMESPKGRp ``` Rules that use selectors refer to IP sets. You can dump the contents of an IP set using the `ipsets` command and you can check whether the IP set contains the expected members: ```text kubectl exec -n calico-system -- calico-node -bpf ipsets dump ``` You can see how many packets have matched each rule. In this example, 9 packets have matched rule `29vMYcPWr7reSxxN` and 0 packets have matched rule `aBMQCbsUMESPKGRp`. Adding `--asm` will show the eBPF assembly code for the program as well. ```text kubectl exec -n calico-system -- calico-node -bpf ipsets dump IP set 0x303904ae5eae5418 192.168.58.5/32 IP set 0xffef9f925a8a4ca4 192.168.129.66/32 192.168.129.68/32 ``` ## Debugging calico-node not ready[​](#debugging-calico-node-not-ready) If you notice that a `calico-node` pod is not ready, check its logs for errors. The most likely reason for `calico-node` not being ready in eBPF mode is that Calico Cloud is not able to update a program attached to an interface. Look for the following type of warning: ```text 2025-09-22 22:39:59.801 [WARNING][10374] felix/bpf_ep_mgr.go 2107: Failed to apply policy to endpoint, leaving it dirty ``` One reason for this type of error is that the eBPF programs provided in the cnx-node image are not compatible with the verifier used by your kernel. Each kernel must ensure that the eBPF programs it loads are safe to run. However, capabilities of the verifier differ between kernel versions. We do test the eBPF programs with a range of kernels, but it is impossible to test all kernels. You may see errors such as if the verifier rejects a program: ```text 265: (79) r1 = *(u64 *)(r10 -72) ; R1_w=ctx(off=0,imm=0) R10=fp0 266: (79) r2 = *(u64 *)(r10 -128) ; R2_w=scalar(umin=14,umax=74,var_off=(0x2; 0x7c)) R10=fp0 267: (b7) r5 = 0 ; R5_w=0 268: (85) call bpf_skb_store_bytes#9 invalid access to map value, value_size=1512 off=8 size=0 R3 min value is outside of the allowed memory range processed 1102 insns (limit 1000000) max_states_per_insn 2 total_states 38 peak_states 38 mark_read 27 -- END PROG LOAD LOG -- libbpf: prog 'calico_tc_skb_ipv4_frag': failed to load: -13 libbpf: failed to load object '/usr/lib/calico/bpf/to_wep_no_log.o' 2025-07-31 17:36:15.708 [WARNING][45] felix/bpf_ep_mgr.go 2124: Failed to apply policy to endpoint, leaving it dirty error=attaching program to wep: loading generic v4 tc hook program: error loading program: error loading object permission denied attaching program to wep: loading generic v4 tc hook program: error loading program: error loading object permission denied name="enif327a56b833" wepID=&types.WorkloadEndpointID{OrchestratorId:"k8s", WorkloadId:"calico-system/csi-node-driver-tggmr", EndpointId:"eth0"} 2025-07-31 17:36:15.708 [WARNING][45] felix/bpf_ep_mgr.go 2124: Failed to apply policy to endpoint, leaving it dirty error=attaching program to wep: loading generic v4 tc hook program: error loading program: error loading object permission denied ``` If you see errors of this type, please open an issue on the Calico Cloud GitHub repository, including details of your kernel version and distribution. ## Poor performance[​](#poor-performance) A number of problems can reduce the performance of the eBPF data plane. - Verify that you are using the best networking mode for your cluster. If possible, avoid using an overlay network; a routed network with no overlay is considerably faster. If you must use one of Calico Cloud's overlay modes, use VXLAN, not IPIP. IPIP performs poorly in eBPF mode due to kernel limitations. - If you are not using an overlay, verify that the [Felix configuration parameters](https://docs.tigera.io/calico-cloud/reference/component-resources/node/felix/configuration) `ipInIpEnabled` and `vxlanEnabled` are set to `false`. Those parameters control whether Felix configured itself to allow IPIP or VXLAN, even if you have no IP pools that use an overlay. The parameters also disable certain eBPF mode optimisations for compatibility with IPIP and VXLAN. To examine the configuration: ```bash kubectl get felixconfiguration -o yaml ``` ```yaml apiVersion: projectcalico.org/v3 items: - apiVersion: projectcalico.org/v3 kind: FelixConfiguration metadata: creationTimestamp: "2020-10-05T13:41:20Z" name: default resourceVersion: "767873" uid: 8df8d751-7449-4b19-a4f9-e33a3d6ccbc0 spec: ... ipipEnabled: false ... vxlanEnabled: false kind: FelixConfigurationList metadata: resourceVersion: "803999" ``` - If you are running your cluster in a cloud such as AWS, then your cloud provider may limit the bandwidth between nodes in your cluster. For example, most AWS nodes are limited to 5GBit per connection. ### Runtime profiling[​](#runtime-profiling) Setting `bpfProfiling` to `Enabled` enables collection of runtime profiling data for eBPF programs. It collects the average execution time and number of executions for each eBPF program attached to each interface. The profiling data can be examined using the `calico-node -bpf profiling e2e` command. The command resets the profiling data after dumping it. ```text ----------------+-------------+-----+-------------+-------+-------------+-------+-------------+-------+ | IFACE | INGRESS NEW | # | INGRESS EST | # | EGRESS NEW | # | EGRESS ETS | # | +----------------+-------------+-----+-------------+-------+-------------+-------+-------------+-------+ | lo | --- | --- | --- | --- | 142.263 ns | 10272 | --- | --- | | eth0 | 2492.344 ns | 32 | 1535.443 ns | 16114 | 6296.421 ns | 749 | 1503.339 ns | 10982 | | eni76136be4c77 | 5031.436 ns | 149 | 1194.923 ns | 1421 | 4950.196 ns | 138 | 1437.015 ns | 1432 | | eni80d5c04bc95 | 7773.459 ns | 74 | 1508.973 ns | 641 | 4907.333 ns | 69 | 1715.848 ns | 646 | | eth1 | 136.250 ns | 24 | --- | --- | 75.320 ns | 25 | --- | --- | | eni5f8ab1cfc29 | 107.250 ns | 36 | 1068.596 ns | 1514 | 189.528 ns | 36 | 1104.335 ns | 1658 | | bpfout.cali | 440.000 ns | 1 | --- | --- | 206.000 ns | 1 | --- | --- | +----------------+-------------+-----+-------------+-------+-------------+-------+-------------+-------+ ``` ## Debug high CPU usage[​](#debug-high-cpu-usage) If you notice `calico-node` using high CPU: - Check if `kube-proxy` is still running. If `kube-proxy` is still running, you must either disable `kube-proxy` or ensure that the Felix configuration setting `bpfKubeProxyIptablesCleanupEnabled` is set to `false`. If the setting is set to `true` (its default), then Felix will attempt to remove `kube-proxy`'s iptables rules. If `kube-proxy` is still running, Felix will continually try to remove the rules, which can cause high CPU usage. - If your cluster is very large, or your workload involves significant service churn, you can increase the interval at which Felix updates the services data plane by increasing the `bpfKubeProxyMinSyncPeriod` setting. The default is 1 second. Increasing the value has the trade-off that service updates will happen more slowly. - Calico Cloud supports endpoint slices, similarly to `kube-proxy`. If your Kubernetes cluster supports endpoint slices and they are enabled, then you can enable endpoint slice support in Calico Cloud with the `bpfKubeProxyEndpointSlicesEnabled` configuration flag. ### Component logs ## Big picture[​](#big-picture) View and collect Calico Cloud logs. ## Value[​](#value) It is useful to view logs to monitor component health and diagnose potential issues. ## Concepts[​](#concepts) ### cnx-node logs[​](#cnx-node-logs) The cnx-node logs contain log output from the following subcomponents: - Per-node startup logic - BGP agent - Felix policy agent Components log either to disk within `/var/log/calico`, to stdout, or both. For components that log to disk, files are automatically rotated, and by default 10 files of 1MB each are kept. The current log file is called `current` and rotated files have @ followed by a timestamp detailing when the files was rotated in [tai64n](http://cr.yp.to/libtai/tai64.html#tai64n) format. ## How to[​](#how-to) ## View logs for a cnx-node instance[​](#view-logs-for-a-cnx-node-instance) You can view logs for a node using the `kubectl logs` command. This will show logs for all subcomponents of the given node. For example: ```text kubectl logs -n calico-system calico-node-xxxx ``` ## View logs from the CNI plugin[​](#view-logs-from-the-cni-plugin) CNI plugin logs are not available through kubectl and are instead logged both to the host machine's disk as well as stderr. By default, these logs can be found at `/var/log/calico/cni/` on the host machine. The container runtime may also display the CNI plugin logs within its own log output. ## Configure BGP agent log level[​](#configure-bgp-agent-log-level) BGP log level is configured via the [BGPConfiguration](https://docs.tigera.io/calico-cloud/reference/resources/bgpconfig) API, and can be one of the following values: - `Debug`: enables "debug all" logging for BIRD. The most verbose logging level. - `Info`: enables logging for protocol state changes. This is the default log level. - `Warning`: disables BIRD logging, emits warning level configuration logs only. - `Error`: disables BIRD logging, emits error level configuration logs only. - `Fatal`: disables BIRD logging, emits fatal level configuration logs only. To modify the BGP log level: 1. Get the current bgpconfig settings. ```bash kubectl get bgpconfiguration.projectcalico.org -o yaml > bgp.yaml ``` 2. Modify logSeverityScreen to the desired value. ```bash vim bgp.yaml ``` > **SUCCESS:** For a global change set the name to "default". For a node-specific change set the name to the node name prefixed with "node.", e.g., "node.node-1". 3. Replace the current bgpconfig settings. ```bash kubectl replace -f bgp.yaml ``` ## Configure Felix log level[​](#configure-felix-log-level) Felix log level is configured via the [FelixConfiguration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig) API, and can be one of the following values: - `Debug`: The most verbose logging level - for development and debugging. - `Info`: The default log level. Shows important state changes. - `Warning`: Shows warnings only. - `Error`: Shows errors only. - `Fatal`: Shows fatal errors only. To modify Felix's log level: 1. Get the current felixconfig settings. ```bash kubectl get felixconfiguration.projectcalico.org default -o yaml > felix.yaml ``` 2. Modify logSeverityScreen to desired value. ```bash vim felixconfig.yaml ``` > **SUCCESS:** For a global change set the name to "default". For a node-specific change set the name to the node name prefixed with "node.", e.g., "node.node-1". 3. Replace the current felixconfig settings. ```text kubectl replace -f felixconfig.yaml ``` --- ## Reference ### Reference APIs, CLI, architecture and design, and FAQ. ## API and installation references[​](#api-and-installation-references) ##### [Tigera Client library](https://docs.tigera.io/calico-cloud/reference/api) [Learn about the Tigera client library and how to use it.](https://docs.tigera.io/calico-cloud/reference/api) ##### [Installation reference](https://docs.tigera.io/calico-cloud/reference/installation/api) [Installation API reference](https://docs.tigera.io/calico-cloud/reference/installation/api) ##### [Image Assurance Installation reference](https://docs.tigera.io/calico-cloud/reference/installation/ia-api) [Image Assurance Installation API reference](https://docs.tigera.io/calico-cloud/reference/installation/ia-api) ##### [REST API Reference](https://docs.tigera.io/calico-cloud/reference/rest-api-reference) [REST API reference](https://docs.tigera.io/calico-cloud/reference/rest-api-reference) ## Resource definitions[​](#resource-definitions) ##### [Resource definitions](https://docs.tigera.io/calico-cloud/reference/resources/overview) [Calico Cloud resources (APIs) that you can manage using calicoctl.](https://docs.tigera.io/calico-cloud/reference/resources/overview) ##### [BFD configuration](https://docs.tigera.io/calico-cloud/reference/resources/bfdconfig) [API for this Calico Enterprise resource.](https://docs.tigera.io/calico-cloud/reference/resources/bfdconfig) ##### [BGP configuration](https://docs.tigera.io/calico-cloud/reference/resources/bgpconfig) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/bgpconfig) ##### [BGP peer](https://docs.tigera.io/calico-cloud/reference/resources/bgppeer) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/bgppeer) ##### [BGP Filter](https://docs.tigera.io/calico-cloud/reference/resources/bgpfilter) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/bgpfilter) ##### [Block affinity](https://docs.tigera.io/calico-cloud/reference/resources/blockaffinity) [IP address management block affinity](https://docs.tigera.io/calico-cloud/reference/resources/blockaffinity) ##### [Calico node status](https://docs.tigera.io/calico-cloud/reference/resources/caliconodestatus) [API for this Calico resource.](https://docs.tigera.io/calico-cloud/reference/resources/caliconodestatus) ##### [Container admission policy](https://docs.tigera.io/calico-cloud/reference/resources/containeradmissionpolicy) [Resource definition.](https://docs.tigera.io/calico-cloud/reference/resources/containeradmissionpolicy) ##### [Compliance reports (deprecated)](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/overview) [Schedule reports and configure report scope.](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/overview) ##### [Inventory report](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/inventory) [API for this resource.](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/inventory) ##### [Network Access report](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/network-access) [API for this resource.](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/network-access) ##### [Policy audit report](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/policy-audit) [API for this resource.](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/policy-audit) ##### [CIS benchmark report](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/cis-benchmark) [API for this resource.](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/cis-benchmark) ##### [Deep packet inspection](https://docs.tigera.io/calico-cloud/reference/resources/deeppacketinspection) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/deeppacketinspection) ##### [Felix configuration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig) [API for this Calico Enterprise resource.](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig) ##### [Egress gateway policy](https://docs.tigera.io/calico-cloud/reference/resources/egressgatewaypolicy) [API for this Calico Enterprise resource.](https://docs.tigera.io/calico-cloud/reference/resources/egressgatewaypolicy) ##### [Global Alert](https://docs.tigera.io/calico-cloud/reference/resources/globalalert) [API for this Calico Enterprise resource.](https://docs.tigera.io/calico-cloud/reference/resources/globalalert) ##### [Global network policy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) ##### [Global network set](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkset) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkset) ##### [Global report](https://docs.tigera.io/calico-cloud/reference/resources/globalreport) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/globalreport) ##### [Global threat feed](https://docs.tigera.io/calico-cloud/reference/resources/globalthreatfeed) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/globalthreatfeed) ##### [Host endpoint](https://docs.tigera.io/calico-cloud/reference/resources/hostendpoint) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/hostendpoint) ##### [IP pool](https://docs.tigera.io/calico-cloud/reference/resources/ippool) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/ippool) ##### [IP reservation](https://docs.tigera.io/calico-cloud/reference/resources/ipreservation) [API for this Calico resource.](https://docs.tigera.io/calico-cloud/reference/resources/ipreservation) ##### [IPAM configuration](https://docs.tigera.io/calico-cloud/reference/resources/ipamconfig) [IP address management global configuration](https://docs.tigera.io/calico-cloud/reference/resources/ipamconfig) ##### [License key](https://docs.tigera.io/calico-cloud/reference/resources/licensekey) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/licensekey) ##### [Kubernetes controllers configuration](https://docs.tigera.io/calico-cloud/reference/resources/kubecontrollersconfig) [API for KubeControllersConfiguration resource.](https://docs.tigera.io/calico-cloud/reference/resources/kubecontrollersconfig) ##### [Managed Cluster](https://docs.tigera.io/calico-cloud/reference/resources/managedcluster) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/managedcluster) ##### [Network policy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) ##### [Network set](https://docs.tigera.io/calico-cloud/reference/resources/networkset) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/networkset) ##### [Node](https://docs.tigera.io/calico-cloud/reference/resources/node) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/node) ##### [PacketCapture](https://docs.tigera.io/calico-cloud/reference/resources/packetcapture) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/packetcapture) ##### [Remote cluster configuration](https://docs.tigera.io/calico-cloud/reference/resources/remoteclusterconfiguration) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/remoteclusterconfiguration) ##### [RuntimeSecurity](https://docs.tigera.io/calico-cloud/reference/resources/runtimesecurity) [API for this Calico Cloud resource](https://docs.tigera.io/calico-cloud/reference/resources/runtimesecurity) ##### [Security event webhook](https://docs.tigera.io/calico-cloud/reference/resources/securityeventwebhook) [API for this Calico Enterprise resource.](https://docs.tigera.io/calico-cloud/reference/resources/securityeventwebhook) ##### [Staged global network policy](https://docs.tigera.io/calico-cloud/reference/resources/stagedglobalnetworkpolicy) [API for this resource.](https://docs.tigera.io/calico-cloud/reference/resources/stagedglobalnetworkpolicy) ##### [Staged Kubernetes network policy](https://docs.tigera.io/calico-cloud/reference/resources/stagedkubernetesnetworkpolicy) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/stagedkubernetesnetworkpolicy) ##### [Staged network policy](https://docs.tigera.io/calico-cloud/reference/resources/stagednetworkpolicy) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/stagednetworkpolicy) ##### [Tier](https://docs.tigera.io/calico-cloud/reference/resources/tier) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/tier) ##### [Workload endpoint](https://docs.tigera.io/calico-cloud/reference/resources/workloadendpoint) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/workloadendpoint) ## Component resources[​](#component-resources) ##### [Configuring the Calico Cloud CNI plugins](https://docs.tigera.io/calico-cloud/reference/component-resources/configuration) [Details for configuring the Calico Cloud CNI plugins.](https://docs.tigera.io/calico-cloud/reference/component-resources/configuration) ##### [Configure resource requests and limits](https://docs.tigera.io/calico-cloud/reference/component-resources/configure-resources) [Configure Resource requests and limits.](https://docs.tigera.io/calico-cloud/reference/component-resources/configure-resources) ##### [Configuring the Calico Cloud Kubernetes controllers](https://docs.tigera.io/calico-cloud/reference/component-resources/kube-controllers/configuration) [Calico Cloud Kubernetes controllers monitor the Kubernetes API and perform actions based on cluster state.](https://docs.tigera.io/calico-cloud/reference/component-resources/kube-controllers/configuration) ##### [Monitoring kube-controllers with Prometheus](https://docs.tigera.io/calico-cloud/reference/component-resources/kube-controllers/prometheus) [Review metrics for the kube-controllers component if you are using Prometheus.](https://docs.tigera.io/calico-cloud/reference/component-resources/kube-controllers/prometheus) ##### [Configuring cnx-node](https://docs.tigera.io/calico-cloud/reference/component-resources/node/configuration) [Customize cnx-node using environment variables.](https://docs.tigera.io/calico-cloud/reference/component-resources/node/configuration) ##### [Configuring Felix](https://docs.tigera.io/calico-cloud/reference/component-resources/node/felix/configuration) [Configure Felix, the daemon that runs on every machine that provides endpoints.](https://docs.tigera.io/calico-cloud/reference/component-resources/node/felix/configuration) ##### [Monitoring Felix with Prometheus](https://docs.tigera.io/calico-cloud/reference/component-resources/node/felix/prometheus) [Review metrics for the Felix component if you are using Prometheus.](https://docs.tigera.io/calico-cloud/reference/component-resources/node/felix/prometheus) ## Configuration on public clouds[​](#configuration-on-public-clouds) ##### [Amazon Web Services](https://docs.tigera.io/calico-cloud/reference/public-cloud/aws) [Advantages of using Calico Cloud in AWS.](https://docs.tigera.io/calico-cloud/reference/public-cloud/aws) ##### [Azure](https://docs.tigera.io/calico-cloud/reference/public-cloud/azure) [Support for Calico Cloud in Azure.](https://docs.tigera.io/calico-cloud/reference/public-cloud/azure) ##### [Google Compute Engine](https://docs.tigera.io/calico-cloud/reference/public-cloud/gce) [Methods to ensure that traffic between containers on different hosts is not dropped by GCE fabric.](https://docs.tigera.io/calico-cloud/reference/public-cloud/gce) ## Host endpoints[​](#host-endpoints) ##### [Host endpoints](https://docs.tigera.io/calico-cloud/reference/host-endpoints/overview) [Secure host network interfaces.](https://docs.tigera.io/calico-cloud/reference/host-endpoints/overview) ##### [Creating policy for basic connectivity](https://docs.tigera.io/calico-cloud/reference/host-endpoints/connectivity) [Customize the Calico failsafe policy to protect host endpoints.](https://docs.tigera.io/calico-cloud/reference/host-endpoints/connectivity) ##### [Creating host endpoint objects](https://docs.tigera.io/calico-cloud/reference/host-endpoints/objects) [To protect a host interface, start by creating a host endpoint object in etcd.](https://docs.tigera.io/calico-cloud/reference/host-endpoints/objects) ##### [Selector-based policies](https://docs.tigera.io/calico-cloud/reference/host-endpoints/selector) [Apply ordered policies to endpoints that match specific label selectors.](https://docs.tigera.io/calico-cloud/reference/host-endpoints/selector) ##### [Failsafe rules](https://docs.tigera.io/calico-cloud/reference/host-endpoints/failsafe) [Avoid cutting off connectivity to hosts because of incorrect network policies.](https://docs.tigera.io/calico-cloud/reference/host-endpoints/failsafe) ##### [Pre-DNAT policy](https://docs.tigera.io/calico-cloud/reference/host-endpoints/pre-dnat) [Apply rules in a host endpoint policy before any DNAT.](https://docs.tigera.io/calico-cloud/reference/host-endpoints/pre-dnat) ##### [Apply on forwarded traffic](https://docs.tigera.io/calico-cloud/reference/host-endpoints/forwarded) [Learn the subtleties using the applyOnForward option in host endpoint policies.](https://docs.tigera.io/calico-cloud/reference/host-endpoints/forwarded) ##### [Summary of host endpoint policies](https://docs.tigera.io/calico-cloud/reference/host-endpoints/summary) [How different host endpoint rules affect packet flows.](https://docs.tigera.io/calico-cloud/reference/host-endpoints/summary) ##### [Connection tracking](https://docs.tigera.io/calico-cloud/reference/host-endpoints/conntrack) [Workaround for Linux conntrack if Calico policy is not working as it should.](https://docs.tigera.io/calico-cloud/reference/host-endpoints/conntrack) ## Architecture[​](#architecture) ##### ['The Calico Cloud data path: IP routing and iptables'](https://docs.tigera.io/calico-cloud/reference/architecture/data-path) [Learn how packets flow between workloads in a datacenter, or between a workload and the internet.](https://docs.tigera.io/calico-cloud/reference/architecture/data-path) ##### [Calico over Ethernet fabrics](https://docs.tigera.io/calico-cloud/reference/architecture/design/l2-interconnect-fabric) [Understand the interconnect fabric options in a Calico network.](https://docs.tigera.io/calico-cloud/reference/architecture/design/l2-interconnect-fabric) ##### [Calico over IP fabrics](https://docs.tigera.io/calico-cloud/reference/architecture/design/l3-interconnect-fabric) [Understand considerations for implementing interconnect fabrics with Calico.](https://docs.tigera.io/calico-cloud/reference/architecture/design/l3-interconnect-fabric) ## Other reference topics[​](#other-reference-topics) ##### [Attribution](https://docs.tigera.io/calico-cloud/reference/attribution) [Attribution report](https://docs.tigera.io/calico-cloud/reference/attribution) ##### [Frequently asked questions](https://docs.tigera.io/calico-cloud/reference/faq) [Common questions that users ask about Calico Enterprise.](https://docs.tigera.io/calico-cloud/reference/faq) ### Tigera Client library Calico Cloud provides and consumes a client library in Go that allows developers to work with Calico Cloud resources. The client library can be used to get, list, watch, create, delete and update custom resources, such as network policies much like the [client-go project](https://github.com/kubernetes/client-go) does for native Kubernetes resources. To learn more about the Tigera client library and how to use it, see the Tigera API project [README](https://github.com/tigera/api/blob/master/README.md) or the [github.com/tigera/api Go module page](https://pkg.go.dev/github.com/tigera/api). ### Installation reference The Kubernetes resources below configure Calico Cloud installation when using the operator. Each resource is responsible for installing and configuring a different subsystem of Calico Cloud during installation. Most options can be modified on a running cluster using `kubectl`. Packages - [operator.tigera.io/v1](#operatortigeraiov1) ## operator.tigera.io/v1[​](#operatortigeraiov1) API Schema definitions for configuring the installation of Calico and Calico Enterprise Package v1 contains API Schema definitions for the operator v1 API group Resource Types - [APIServer](#apiserver) - [ApplicationLayer](#applicationlayer) - [Authentication](#authentication) - [Compliance](#compliance) - [EgressGateway](#egressgateway) - [GatewayAPI](#gatewayapi) - [Goldmane](#goldmane) - [ImageSet](#imageset) - [Installation](#installation) - [IntrusionDetection](#intrusiondetection) - [LogCollector](#logcollector) - [LogStorage](#logstorage) - [ManagementCluster](#managementcluster) - [ManagementClusterConnection](#managementclusterconnection) - [Manager](#manager) - [Monitor](#monitor) - [NonClusterHost](#nonclusterhost) - [PacketCaptureAPI](#packetcaptureapi) - [PolicyRecommendation](#policyrecommendation) - [TigeraStatus](#tigerastatus) - [Whisker](#whisker) ### APIServer[​](#apiserver) APIServer installs the Tigera API server and related resources. At most one instance of this resource is supported. It must be named "default" or "tigera-secure". | Field | Description | | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------- | | `apiVersion` *string* | `operator.tigera.io/v1` | | `kind` *string* | `APIServer` | | `metadata` *[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#objectmeta-v1-meta)* | Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[APIServerSpec](#apiserverspec)* | Specification of the desired state for the Tigera API server. | | `status` *[APIServerStatus](#apiserverstatus)* | Most recently observed status for the Tigera API server. | ### APIServerDeployment[​](#apiserverdeployment) APIServerDeployment is the configuration for the API server Deployment. *Appears in:* - [APIServerSpec](#apiserverspec) | Field | Description | | ------------------------------------------------------------ | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[APIServerDeploymentSpec](#apiserverdeploymentspec)* | (Optional) Spec is the specification of the API server Deployment. | ### APIServerDeploymentContainer[​](#apiserverdeploymentcontainer) APIServerDeploymentContainer is an API server Deployment container. *Appears in:* - [APIServerDeploymentPodSpec](#apiserverdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the API server Deployment container by name. Supported values are: calico-apiserver, tigera-queryserver, calico-l7-admission-controller | | `ports` *[APIServerDeploymentContainerPort](#apiserverdeploymentcontainerport) array* | (Optional) Ports allows customization of container's ports. If specified, this overrides the named APIServer Deployment container's ports. If omitted, the API server Deployment will use its default value for this container's port. | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named API server Deployment container's resources. If omitted, the API server Deployment will use its default value for this container's resources. If used in conjunction with the deprecated ComponentResources, then this value takes precedence. | ### APIServerDeploymentContainerPort[​](#apiserverdeploymentcontainerport) *Appears in:* - [APIServerDeploymentContainer](#apiserverdeploymentcontainer) | Field | Description | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `name` *string* | Name is an enum which identifies the API server Deployment Container port by name. Supported values are: apiserver, queryserver, l7admctrl | | `containerPort` *integer* | Number of port to expose on the pod's IP address. This must be a valid port number, 0 < x < 65536. | ### APIServerDeploymentInitContainer[​](#apiserverdeploymentinitcontainer) APIServerDeploymentInitContainer is an API server Deployment init container. *Appears in:* - [APIServerDeploymentPodSpec](#apiserverdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the API server Deployment init container by name. Supported values are: calico-apiserver-certs-key-cert-provisioner | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named API server Deployment init container's resources. If omitted, the API server Deployment will use its default value for this init container's resources. | ### APIServerDeploymentPodSpec[​](#apiserverdeploymentpodspec) APIServerDeploymentDeploymentPodSpec is the API server Deployment's PodSpec. *Appears in:* - [APIServerDeploymentPodTemplateSpec](#apiserverdeploymentpodtemplatespec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[APIServerDeploymentInitContainer](#apiserverdeploymentinitcontainer) array* | (Optional) InitContainers is a list of API server init containers. If specified, this overrides the specified API server Deployment init containers. If omitted, the API server Deployment will use its default values for its init containers. | | `containers` *[APIServerDeploymentContainer](#apiserverdeploymentcontainer) array* | (Optional) Containers is a list of API server containers. If specified, this overrides the specified API server Deployment containers. If omitted, the API server Deployment will use its default values for its containers. | | `affinity` *[Affinity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#affinity-v1-core)* | (Optional) Affinity is a group of affinity scheduling rules for the API server pods. If specified, this overrides any affinity that may be set on the API server Deployment. If omitted, the API server Deployment will use its default value for affinity. WARNING: Please note that this field will override the default API server Deployment affinity. | | `nodeSelector` *object (keys:string, values:string)* | NodeSelector is the API server pod's scheduling constraints. If specified, each of the key/value pairs are added to the API server Deployment nodeSelector provided the key does not already exist in the object's nodeSelector. If used in conjunction with ControlPlaneNodeSelector, that nodeSelector is set on the API server Deployment and each of this field's key/value pairs are added to the API server Deployment nodeSelector provided the key does not already exist in the object's nodeSelector. If omitted, the API server Deployment will use its default value for nodeSelector. WARNING: Please note that this field will modify the default API server Deployment nodeSelector. | | `topologySpreadConstraints` *[TopologySpreadConstraint](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#topologyspreadconstraint-v1-core) array* | (Optional) TopologySpreadConstraints describes how a group of pods ought to spread across topology domains. Scheduler will schedule pods in a way which abides by the constraints. All topologySpreadConstraints are ANDed. | | `tolerations` *[Toleration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#toleration-v1-core) array* | (Optional) Tolerations is the API server pod's tolerations. If specified, this overrides any tolerations that may be set on the API server Deployment. If omitted, the API server Deployment will use its default value for tolerations. WARNING: Please note that this field will override the default API server Deployment tolerations. | | `priorityClassName` *string* | (Optional) PriorityClassName allows to specify a PriorityClass resource to be used. | ### APIServerDeploymentPodTemplateSpec[​](#apiserverdeploymentpodtemplatespec) APIServerDeploymentPodTemplateSpec is the API server Deployment's PodTemplateSpec *Appears in:* - [APIServerDeploymentSpec](#apiserverdeploymentspec) | Field | Description | | ------------------------------------------------------------------ | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[APIServerDeploymentPodSpec](#apiserverdeploymentpodspec)* | (Optional) Spec is the API server Deployment's PodSpec. | ### APIServerDeploymentSpec[​](#apiserverdeploymentspec) APIServerDeploymentSpec defines configuration for the API server Deployment. *Appears in:* - [APIServerDeployment](#apiserverdeployment) | Field | Description | | -------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `minReadySeconds` *integer* | (Optional) MinReadySeconds is the minimum number of seconds for which a newly created Deployment pod should be ready without any of its container crashing, for it to be considered available. If specified, this overrides any minReadySeconds value that may be set on the API server Deployment. If omitted, the API server Deployment will use its default value for minReadySeconds. | | `template` *[APIServerDeploymentPodTemplateSpec](#apiserverdeploymentpodtemplatespec)* | (Optional) Template describes the API server Deployment pod that will be created. | ### APIServerLogging[​](#apiserverlogging) *Appears in:* - [APIServerPodLogging](#apiserverpodlogging) | Field | Description | | ------------------------------------------- | ----------------------------------------------------------------- | | `logSeverity` *[LogSeverity](#logseverity)* | (Optional) LogSeverity defines log level for APIServer container. | ### APIServerPodLogging[​](#apiserverpodlogging) *Appears in:* - [APIServerSpec](#apiserverspec) | Field | Description | | --------------------------------------------------------- | ----------- | | `apiServer` *[APIServerLogging](#apiserverlogging)* | (Optional) | | `queryServer` *[QueryServerLogging](#queryserverlogging)* | (Optional) | ### APIServerSpec[​](#apiserverspec) APIServerSpec defines the desired state of Tigera API server. *Appears in:* - [APIServer](#apiserver) | Field | Description | | ------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `logging` *[APIServerPodLogging](#apiserverpodlogging)* | (Optional) | | `apiServerDeployment` *[APIServerDeployment](#apiserverdeployment)* | APIServerDeployment configures the calico-apiserver Deployment. If used in conjunction with ControlPlaneNodeSelector or ControlPlaneTolerations, then these overrides take precedence. | ### APIServerStatus[​](#apiserverstatus) APIServerStatus defines the observed state of Tigera API server. *Appears in:* - [APIServer](#apiserver) | Field | Description | | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `state` *string* | State provides user-readable status. | | `conditions` *[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#condition-v1-meta) array* | (Optional) Conditions represents the latest observed set of conditions for the component. A component may be one or more of Ready, Progressing, Degraded or other customer types. | ### AWSEgressGateway[​](#awsegressgateway) AWSEgressGateway defines the configurations for deploying EgressGateway in AWS *Appears in:* - [EgressGatewaySpec](#egressgatewayspec) | Field | Description | | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `nativeIP` *[NativeIP](#nativeip)* | (Optional) NativeIP defines if EgressGateway is to use an AWS backed IPPool. Default: Disabled | | `elasticIPs` *string array* | (Optional) ElasticIPs defines the set of elastic IPs that can be used for Egress Gateway pods. NativeIP must be Enabled if elastic IPs are set. | ### AdditionalLogSourceSpec[​](#additionallogsourcespec) *Appears in:* - [LogCollectorSpec](#logcollectorspec) | Field | Description | | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | | `eksCloudwatchLog` *[EksCloudwatchLogsSpec](#ekscloudwatchlogsspec)* | (Optional) If specified with EKS Provider in Installation, enables fetching EKS audit logs. | ### AdditionalLogStoreSpec[​](#additionallogstorespec) *Appears in:* - [LogCollectorSpec](#logcollectorspec) | Field | Description | | ---------------------------------------------- | --------------------------------------------------------------------------------------------- | | `s3` *[S3StoreSpec](#s3storespec)* | (Optional) If specified, enables exporting of flow, audit, and DNS logs to Amazon S3 storage. | | `syslog` *[SyslogStoreSpec](#syslogstorespec)* | (Optional) If specified, enables exporting of flow, audit, and DNS logs to syslog. | | `splunk` *[SplunkStoreSpec](#splunkstorespec)* | (Optional) If specified, enables exporting of flow, audit, and DNS logs to splunk. | ### AlertManager[​](#alertmanager) *Appears in:* - [MonitorSpec](#monitorspec) | Field | Description | | ---------------------------------------------- | --------------------------------------------------------- | | `spec` *[AlertManagerSpec](#alertmanagerspec)* | (Optional) Spec is the specification of the Alertmanager. | ### AlertManagerSpec[​](#alertmanagerspec) *Appears in:* - [AlertManager](#alertmanager) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | Define resources requests and limits for single Pods. | ### AnomalyDetectionSpec[​](#anomalydetectionspec) *Appears in:* - [IntrusionDetectionSpec](#intrusiondetectionspec) | Field | Description | | --------------------------- | -------------------------------------------------------------------------------- | | `storageClassName` *string* | (Optional) StorageClassName is now deprecated, and configuring it has no effect. | ### ApplicationLayer[​](#applicationlayer) ApplicationLayer is the Schema for the applicationlayers API | Field | Description | | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------- | | `apiVersion` *string* | `operator.tigera.io/v1` | | `kind` *string* | `ApplicationLayer` | | `metadata` *[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#objectmeta-v1-meta)* | Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[ApplicationLayerSpec](#applicationlayerspec)* | | | `status` *[ApplicationLayerStatus](#applicationlayerstatus)* | | ### ApplicationLayerPolicyStatusType[​](#applicationlayerpolicystatustype) *Underlying type:* *string* *Validation:* - Enum: \[Enabled Disabled] *Appears in:* - [ApplicationLayerSpec](#applicationlayerspec) | Value | Description | | ---------- | ----------- | | `Enabled` | | | `Disabled` | | ### ApplicationLayerSpec[​](#applicationlayerspec) ApplicationLayerSpec defines the desired state of ApplicationLayer *Appears in:* - [ApplicationLayer](#applicationlayer) | Field | Description | | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `webApplicationFirewall` *[WAFStatusType](#wafstatustype)* | WebApplicationFirewall controls whether or not ModSecurity enforcement is enabled for the cluster. When enabled, Services may opt-in to having ingress traffic examed by ModSecurity. | | `logCollection` *[LogCollectionSpec](#logcollectionspec)* | Specification for application layer (L7) log collection. | | `applicationLayerPolicy` *[ApplicationLayerPolicyStatusType](#applicationlayerpolicystatustype)* | Application Layer Policy controls whether or not ALP enforcement is enabled for the cluster. When enabled, NetworkPolicies with HTTP Match rules may be defined to opt-in workloads for traffic enforcement on the application layer. | | `envoy` *[EnvoySettings](#envoysettings)* | User-configurable settings for the Envoy proxy. | | `l7LogCollectorDaemonSet` *[L7LogCollectorDaemonSet](#l7logcollectordaemonset)* | (Optional) L7LogCollectorDaemonSet configures the L7LogCollector DaemonSet. | | `sidecarInjection` *[SidecarStatusType](#sidecarstatustype)* | (Optional) SidecarInjection controls whether or not sidecar injection is enabled for the cluster. When enabled, pods with the label "applicationlayer.projectcalico.org/sidecar"="true" will have their L7 functionality such as WAF and ALP implemented using an injected sidecar instead of a per-host proxy. The per-host proxy will continue to be used for pods without this label. | ### ApplicationLayerStatus[​](#applicationlayerstatus) ApplicationLayerStatus defines the observed state of ApplicationLayer *Appears in:* - [ApplicationLayer](#applicationlayer) | Field | Description | | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `state` *string* | State provides user-readable status. | | `sidecarWebhook` *[SidecarWebhookStateType](#sidecarwebhookstatetype)* | SidecarWebhook provides the state of sidecar injection mutatinwebhookconfiguration | | `conditions` *[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#condition-v1-meta) array* | (Optional) Conditions represents the latest observed set of conditions for the component. A component may be one or more of Ready, Progressing, Degraded or other customer types. | ### AssignmentMode[​](#assignmentmode) *Underlying type:* *string* *Appears in:* - [IPPool](#ippool) | Value | Description | | ----------- | ----------- | | `Automatic` | | | `Manual` | | ### Authentication[​](#authentication) Authentication is the Schema for the authentications API | Field | Description | | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------- | | `apiVersion` *string* | `operator.tigera.io/v1` | | `kind` *string* | `Authentication` | | `metadata` *[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#objectmeta-v1-meta)* | Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[AuthenticationSpec](#authenticationspec)* | | | `status` *[AuthenticationStatus](#authenticationstatus)* | | ### AuthenticationLDAP[​](#authenticationldap) AuthenticationLDAP is the configuration needed to setup LDAP. *Appears in:* - [AuthenticationSpec](#authenticationspec) | Field | Description | | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `host` *string* | The host and port of the LDAP server. Example: ad.example.com:636 | | `startTLS` *boolean* | (Optional) StartTLS whether to enable the startTLS feature for establishing TLS on an existing LDAP session. If true, the ldap\:// protocol is used and then issues a StartTLS command, otherwise, connections will use the ldaps\:// protocol. | | `userSearch` *[UserSearch](#usersearch)* | User entry search configuration to match the credentials with a user. | | `groupSearch` *[GroupSearch](#groupsearch)* | (Optional) Group search configuration to find the groups that a user is in. | ### AuthenticationOIDC[​](#authenticationoidc) AuthenticationOIDC is the configuration needed to setup OIDC. *Appears in:* - [AuthenticationSpec](#authenticationspec) | Field | Description | | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `issuerURL` *string* | IssuerURL is the URL to the OIDC provider. | | `usernameClaim` *string* | UsernameClaim specifies which claim to use from the OIDC provider as the username. | | `requestedScopes` *string array* | (Optional) RequestedScopes is a list of scopes to request from the OIDC provider. If not provided, the following scopes are requested: \["openid", "email", "profile", "groups", "offline\_access"]. | | `usernamePrefix` *string* | (Optional) Deprecated. Please use Authentication.Spec.UsernamePrefix instead. | | `groupsClaim` *string* | (Optional) GroupsClaim specifies which claim to use from the OIDC provider as the group. | | `groupsPrefix` *string* | (Optional) Deprecated. Please use Authentication.Spec.GroupsPrefix instead. | | `emailVerification` *[EmailVerificationType](#emailverificationtype)* | (Optional) Some providers do not include the claim "email\_verified" when there is no verification in the user enrollment process or if they are acting as a proxy for another identity provider. By default those tokens are deemed invalid. To skip this check, set the value to "InsecureSkip". Default: Verify | | `promptTypes` *[PromptType](#prompttype) array* | (Optional) PromptTypes is an optional list of string values that specifies whether the identity provider prompts the end user for re-authentication and consent. See the RFC for more information on prompt types: [https://openid.net/specs/openid-connect-core-1\_0.html](https://openid.net/specs/openid-connect-core-1_0.html). Default: "Consent" | | `type` *[OIDCType](#oidctype)* | (Optional) Default: "Dex" | ### AuthenticationOpenshift[​](#authenticationopenshift) AuthenticationOpenshift is the configuration needed to setup Openshift. *Appears in:* - [AuthenticationSpec](#authenticationspec) | Field | Description | | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | | `issuerURL` *string* | IssuerURL is the URL to the Openshift OAuth provider. Ex.: [https://api.my-ocp-domain.com:6443](https://api.my-ocp-domain.com:6443) | ### AuthenticationSpec[​](#authenticationspec) AuthenticationSpec defines the desired state of Authentication *Appears in:* - [Authentication](#authentication) | Field | Description | | ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `managerDomain` *string* | ManagerDomain is the domain name of the Manager | | `usernamePrefix` *string* | (Optional) If specified, UsernamePrefix is prepended to each user obtained from the identity provider. Note that Kibana does not support a user prefix, so this prefix is removed from Kubernetes User when translating log access ClusterRoleBindings into Elastic. | | `groupsPrefix` *string* | (Optional) If specified, GroupsPrefix is prepended to each group obtained from the identity provider. Note that Kibana does not support a groups prefix, so this prefix is removed from Kubernetes Groups when translating log access ClusterRoleBindings into Elastic. | | `oidc` *[AuthenticationOIDC](#authenticationoidc)* | (Optional) OIDC contains the configuration needed to setup OIDC authentication. | | `openshift` *[AuthenticationOpenshift](#authenticationopenshift)* | (Optional) Openshift contains the configuration needed to setup Openshift OAuth authentication. | | `ldap` *[AuthenticationLDAP](#authenticationldap)* | (Optional) LDAP contains the configuration needed to setup LDAP authentication. | | `dexDeployment` *[DexDeployment](#dexdeployment)* | (Optional) DexDeployment configures the Dex Deployment. | ### AuthenticationStatus[​](#authenticationstatus) AuthenticationStatus defines the observed state of Authentication *Appears in:* - [Authentication](#authentication) | Field | Description | | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `state` *string* | State provides user-readable status. | | `conditions` *[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#condition-v1-meta) array* | (Optional) Conditions represents the latest observed set of conditions for the component. A component may be one or more of Ready, Progressing, Degraded or other customer types. | ### Azure[​](#azure) *Appears in:* - [InstallationSpec](#installationspec) | Field | Description | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `policyMode` *[PolicyMode](#policymode)* | (Optional) PolicyMode determines whether the "control-plane" label is applied to namespaces. It offers two options: Default and Manual. The Default option adds the "control-plane" label to the required namespaces. The Manual option does not apply the "control-plane" label to any namespace. Default: Default | ### BGPOption[​](#bgpoption) *Underlying type:* *string* BGPOption describes the mode of BGP to use. One of: Enabled, Disabled *Appears in:* - [CalicoNetworkSpec](#caliconetworkspec) | Value | Description | | ---------- | ----------- | | `Enabled` | | | `Disabled` | | ### BPFNetworkBootstrapType[​](#bpfnetworkbootstraptype) *Underlying type:* *string* BPFNetworkBootstrapType defines how the initial networking configuration is executed. *Appears in:* - [CalicoNetworkSpec](#caliconetworkspec) | Value | Description | | ---------- | ----------- | | `Enabled` | | | `Disabled` | | ### CAType[​](#catype) *Underlying type:* *string* CAType specifies which verification method the tunnel client should use to verify the tunnel server's identity. One of: Tigera, Public *Appears in:* - [ManagementClusterTLS](#managementclustertls) | Value | Description | | -------- | ----------- | | `Tigera` | | | `Public` | | ### CNILogging[​](#cnilogging) *Appears in:* - [Logging](#logging) | Field | Description | | ------------------------------------------------------------------------------------------------------------------------- | ----------------------------- | | `logSeverity` *[LogLevel](#loglevel)* | (Optional) Default: Info | | `logFileMaxSize` *[Quantity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#quantity-resource-api)* | (Optional) Default: 100Mi | | `logFileMaxAgeDays` *integer* | (Optional) Default: 30 (days) | | `logFileMaxCount` *integer* | (Optional) Default: 10 | ### CNIPluginType[​](#cniplugintype) *Underlying type:* *string* CNIPluginType describes the type of CNI plugin used. One of: Calico, GKE, AmazonVPC, AzureVNET *Appears in:* - [CNISpec](#cnispec) | Value | Description | | ----------- | ----------- | | `Calico` | | | `GKE` | | | `AmazonVPC` | | | `AzureVNET` | | ### CNISpec[​](#cnispec) CNISpec contains configuration for the CNI plugin. *Appears in:* - [InstallationSpec](#installationspec) | Field | Description | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` *[CNIPluginType](#cniplugintype)* | Specifies the CNI plugin that will be used in the Calico or Calico Enterprise installation. \* For KubernetesProvider GKE, this field defaults to GKE. \* For KubernetesProvider AKS, this field defaults to AzureVNET. \* For KubernetesProvider EKS, this field defaults to AmazonVPC. \* If aws-node daemonset exists in kube-system when the Installation resource is created, this field defaults to AmazonVPC. \* For all other cases this field defaults to Calico. For the value Calico, the CNI plugin binaries and CNI config will be installed as part of deployment, for all other values the CNI plugin binaries and CNI config is a dependency that is expected to be installed separately. Default: Calico | | `ipam` *[IPAMSpec](#ipamspec)* | (Optional) IPAM specifies the pod IP address management that will be used in the Calico or Calico Enterprise installation. | | `binDir` *string* | (Optional) BinDir is the path to the CNI binaries directory. If you have changed the installation directory for CNI binaries in the container runtime configuration, please ensure that this field points to the same directory as specified in the container runtime settings. Default directory depends on the KubernetesProvider. \* For KubernetesProvider GKE, this field defaults to "/home/kubernetes/bin". \* For KubernetesProvider OpenShift, this field defaults to "/var/lib/cni/bin". \* Otherwise, this field defaults to "/opt/cni/bin". | | `confDir` *string* | (Optional) ConfDir is the path to the CNI config directory. If you have changed the installation directory for CNI configuration in the container runtime configuration, please ensure that this field points to the same directory as specified in the container runtime settings. Default directory depends on the KubernetesProvider. \* For KubernetesProvider GKE, this field defaults to "/etc/cni/net.d". \* For KubernetesProvider OpenShift, this field defaults to "/var/run/multus/cni/net.d". \* Otherwise, this field defaults to "/etc/cni/net.d". | ### CRDManagement[​](#crdmanagement) *Underlying type:* *string* *Validation:* - Enum: \[Reconcile PreferExisting] *Appears in:* - [GatewayAPISpec](#gatewayapispec) | Value | Description | | ---------------- | ----------- | | `Reconcile` | | | `PreferExisting` | | ### CSINodeDriverDaemonSet[​](#csinodedriverdaemonset) CSINodeDriverDaemonSet is the configuration for the csi-node-driver DaemonSet. *Appears in:* - [InstallationSpec](#installationspec) | Field | Description | | ------------------------------------------------------------------ | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[CSINodeDriverDaemonSetSpec](#csinodedriverdaemonsetspec)* | (Optional) Spec is the specification of the csi-node-driver DaemonSet. | ### CSINodeDriverDaemonSetContainer[​](#csinodedriverdaemonsetcontainer) CSINodeDriverDaemonSetContainer is a csi-node-driver DaemonSet container. *Appears in:* - [CSINodeDriverDaemonSetPodSpec](#csinodedriverdaemonsetpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the csi-node-driver DaemonSet container by name. Supported values are: calico-csi, csi-node-driver-registrar. | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named csi-node-driver DaemonSet container's resources. If omitted, the csi-node-driver DaemonSet will use its default value for this container's resources. | ### CSINodeDriverDaemonSetPodSpec[​](#csinodedriverdaemonsetpodspec) CSINodeDriverDaemonSetPodSpec is the csi-node-driver DaemonSet's PodSpec. *Appears in:* - [CSINodeDriverDaemonSetPodTemplateSpec](#csinodedriverdaemonsetpodtemplatespec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `containers` *[CSINodeDriverDaemonSetContainer](#csinodedriverdaemonsetcontainer) array* | (Optional) Containers is a list of csi-node-driver containers. If specified, this overrides the specified csi-node-driver DaemonSet containers. If omitted, the csi-node-driver DaemonSet will use its default values for its containers. | | `affinity` *[Affinity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#affinity-v1-core)* | (Optional) Affinity is a group of affinity scheduling rules for the csi-node-driver pods. If specified, this overrides any affinity that may be set on the csi-node-driver DaemonSet. If omitted, the csi-node-driver DaemonSet will use its default value for affinity. WARNING: Please note that this field will override the default csi-node-driver DaemonSet affinity. | | `nodeSelector` *object (keys:string, values:string)* | (Optional) NodeSelector is the csi-node-driver pod's scheduling constraints. If specified, each of the key/value pairs are added to the csi-node-driver DaemonSet nodeSelector provided the key does not already exist in the object's nodeSelector. If omitted, the csi-node-driver DaemonSet will use its default value for nodeSelector. WARNING: Please note that this field will modify the default csi-node-driver DaemonSet nodeSelector. | | `tolerations` *[Toleration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#toleration-v1-core) array* | (Optional) Tolerations is the csi-node-driver pod's tolerations. If specified, this overrides any tolerations that may be set on the csi-node-driver DaemonSet. If omitted, the csi-node-driver DaemonSet will use its default value for tolerations. WARNING: Please note that this field will override the default csi-node-driver DaemonSet tolerations. | ### CSINodeDriverDaemonSetPodTemplateSpec[​](#csinodedriverdaemonsetpodtemplatespec) CSINodeDriverDaemonSetPodTemplateSpec is the csi-node-driver DaemonSet's PodTemplateSpec *Appears in:* - [CSINodeDriverDaemonSetSpec](#csinodedriverdaemonsetspec) | Field | Description | | ------------------------------------------------------------------------ | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[CSINodeDriverDaemonSetPodSpec](#csinodedriverdaemonsetpodspec)* | (Optional) Spec is the csi-node-driver DaemonSet's PodSpec. | ### CSINodeDriverDaemonSetSpec[​](#csinodedriverdaemonsetspec) CSINodeDriverDaemonSetSpec defines configuration for the csi-node-driver DaemonSet. *Appears in:* - [CSINodeDriverDaemonSet](#csinodedriverdaemonset) | Field | Description | | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `minReadySeconds` *integer* | (Optional) MinReadySeconds is the minimum number of seconds for which a newly created DaemonSet pod should be ready without any of its container crashing, for it to be considered available. If specified, this overrides any minReadySeconds value that may be set on the csi-node-driver DaemonSet. If omitted, the csi-node-driver DaemonSet will use its default value for minReadySeconds. | | `template` *[CSINodeDriverDaemonSetPodTemplateSpec](#csinodedriverdaemonsetpodtemplatespec)* | (Optional) Template describes the csi-node-driver DaemonSet pod that will be created. | ### CalicoKubeControllersDeployment[​](#calicokubecontrollersdeployment) CalicoKubeControllersDeployment is the configuration for the calico-kube-controllers Deployment. *Appears in:* - [InstallationSpec](#installationspec) | Field | Description | | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[CalicoKubeControllersDeploymentSpec](#calicokubecontrollersdeploymentspec)* | (Optional) Spec is the specification of the calico-kube-controllers Deployment. | ### CalicoKubeControllersDeploymentContainer[​](#calicokubecontrollersdeploymentcontainer) CalicoKubeControllersDeploymentContainer is a calico-kube-controllers Deployment container. *Appears in:* - [CalicoKubeControllersDeploymentPodSpec](#calicokubecontrollersdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `name` *string* | Name is an enum which identifies the calico-kube-controllers Deployment container by name. Supported values are: calico-kube-controllers, es-calico-kube-controllers | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named calico-kube-controllers Deployment container's resources. If omitted, the calico-kube-controllers Deployment will use its default value for this container's resources. If used in conjunction with the deprecated ComponentResources, then this value takes precedence. | ### CalicoKubeControllersDeploymentPodSpec[​](#calicokubecontrollersdeploymentpodspec) CalicoKubeControllersDeploymentPodSpec is the calico-kube-controller Deployment's PodSpec. *Appears in:* - [CalicoKubeControllersDeploymentPodTemplateSpec](#calicokubecontrollersdeploymentpodtemplatespec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `containers` *[CalicoKubeControllersDeploymentContainer](#calicokubecontrollersdeploymentcontainer) array* | (Optional) Containers is a list of calico-kube-controllers containers. If specified, this overrides the specified calico-kube-controllers Deployment containers. If omitted, the calico-kube-controllers Deployment will use its default values for its containers. | | `affinity` *[Affinity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#affinity-v1-core)* | (Optional) Affinity is a group of affinity scheduling rules for the calico-kube-controllers pods. If specified, this overrides any affinity that may be set on the calico-kube-controllers Deployment. If omitted, the calico-kube-controllers Deployment will use its default value for affinity. WARNING: Please note that this field will override the default calico-kube-controllers Deployment affinity. | | `nodeSelector` *object (keys:string, values:string)* | NodeSelector is the calico-kube-controllers pod's scheduling constraints. If specified, each of the key/value pairs are added to the calico-kube-controllers Deployment nodeSelector provided the key does not already exist in the object's nodeSelector. If used in conjunction with ControlPlaneNodeSelector, that nodeSelector is set on the calico-kube-controllers Deployment and each of this field's key/value pairs are added to the calico-kube-controllers Deployment nodeSelector provided the key does not already exist in the object's nodeSelector. If omitted, the calico-kube-controllers Deployment will use its default value for nodeSelector. WARNING: Please note that this field will modify the default calico-kube-controllers Deployment nodeSelector. | | `tolerations` *[Toleration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#toleration-v1-core) array* | (Optional) Tolerations is the calico-kube-controllers pod's tolerations. If specified, this overrides any tolerations that may be set on the calico-kube-controllers Deployment. If omitted, the calico-kube-controllers Deployment will use its default value for tolerations. WARNING: Please note that this field will override the default calico-kube-controllers Deployment tolerations. | ### CalicoKubeControllersDeploymentPodTemplateSpec[​](#calicokubecontrollersdeploymentpodtemplatespec) CalicoKubeControllersDeploymentPodTemplateSpec is the calico-kube-controllers Deployment's PodTemplateSpec *Appears in:* - [CalicoKubeControllersDeploymentSpec](#calicokubecontrollersdeploymentspec) | Field | Description | | ------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[CalicoKubeControllersDeploymentPodSpec](#calicokubecontrollersdeploymentpodspec)* | (Optional) Spec is the calico-kube-controllers Deployment's PodSpec. | ### CalicoKubeControllersDeploymentSpec[​](#calicokubecontrollersdeploymentspec) CalicoKubeControllersDeploymentSpec defines configuration for the calico-kube-controllers Deployment. *Appears in:* - [CalicoKubeControllersDeployment](#calicokubecontrollersdeployment) | Field | Description | | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `minReadySeconds` *integer* | (Optional) MinReadySeconds is the minimum number of seconds for which a newly created Deployment pod should be ready without any of its container crashing, for it to be considered available. If specified, this overrides any minReadySeconds value that may be set on the calico-kube-controllers Deployment. If omitted, the calico-kube-controllers Deployment will use its default value for minReadySeconds. | | `template` *[CalicoKubeControllersDeploymentPodTemplateSpec](#calicokubecontrollersdeploymentpodtemplatespec)* | (Optional) Template describes the calico-kube-controllers Deployment pod that will be created. | ### CalicoNetworkSpec[​](#caliconetworkspec) CalicoNetworkSpec specifies configuration options for Calico provided pod networking. *Appears in:* - [InstallationSpec](#installationspec) | Field | Description | | ------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `linuxDataplane` *[LinuxDataplaneOption](#linuxdataplaneoption)* | (Optional) LinuxDataplane is used to select the dataplane used for Linux nodes. In particular, it causes the operator to add required mounts and environment variables for the particular dataplane. If not specified, iptables mode is used. Default: Iptables | | `windowsDataplane` *[WindowsDataplaneOption](#windowsdataplaneoption)* | (Optional) WindowsDataplane is used to select the dataplane used for Windows nodes. In particular, it causes the operator to add required mounts and environment variables for the particular dataplane. If not specified, it is disabled and the operator will not render the Calico Windows nodes daemonset. Default: Disabled | | `bpfNetworkBootstrap` *[BPFNetworkBootstrapType](#bpfnetworkbootstraptype)* | (Optional) BPFNetworkBootstrap manages the initial networking setup required to configure the BPF dataplane. When enabled, the operator tries to bootstraps access to the Kubernetes API Server by using the Kubernetes service and its associated endpoints. This field should be enabled only if linuxDataplane is set to "BPF". If another dataplane is selected, this field must be omitted or explicitly set to Disabled. When disabled and linuxDataplane is BPF, you must manually provide the Kubernetes API Server information via the "kubernetes-service-endpoint" ConfigMap. It is invalid to use both the ConfigMap and have this field set to true at the same time. Default: Disabled | | `kubeProxyManagement` *[KubeProxyManagementType](#kubeproxymanagementtype)* | (Optional) KubeProxyManagement controls whether the operator manages the kube-proxy DaemonSet. When enabled, the operator will manage the DaemonSet by patching it: it disables kube-proxy if the dataplane is BPF, or enables it otherwise. Default: Disabled | | `bgp` *[BGPOption](#bgpoption)* | (Optional) BGP configures whether or not to enable Calico's BGP capabilities. | | `ipPools` *[IPPool](#ippool) array* | (Optional) IPPools contains a list of IP pools to manage. If nil, a single IPv4 IP pool will be created by the operator. If an empty list is provided, the operator will not create any IP pools and will instead wait for IP pools to be created out-of-band. IP pools in this list will be reconciled by the operator and should not be modified out-of-band. | | `mtu` *integer* | (Optional) MTU specifies the maximum transmission unit to use on the pod network. If not specified, Calico will perform MTU auto-detection based on the cluster network. | | `nodeAddressAutodetectionV4` *[NodeAddressAutodetection](#nodeaddressautodetection)* | (Optional) NodeAddressAutodetectionV4 specifies an approach to automatically detect node IPv4 addresses. If not specified, will use default auto-detection settings to acquire an IPv4 address for each node. | | `nodeAddressAutodetectionV6` *[NodeAddressAutodetection](#nodeaddressautodetection)* | (Optional) NodeAddressAutodetectionV6 specifies an approach to automatically detect node IPv6 addresses. If not specified, IPv6 addresses will not be auto-detected. | | `hostPorts` *[HostPortsType](#hostportstype)* | (Optional) HostPorts configures whether or not Calico will support Kubernetes HostPorts. Valid only when using the Calico CNI plugin. Default: Enabled | | `multiInterfaceMode` *[MultiInterfaceMode](#multiinterfacemode)* | (Optional) MultiInterfaceMode configures what will configure multiple interface per pod. Only valid for Calico Enterprise installations using the Calico CNI plugin. Default: None | | `containerIPForwarding` *[ContainerIPForwardingType](#containeripforwardingtype)* | (Optional) ContainerIPForwarding configures whether ip forwarding will be enabled for containers in the CNI configuration. Default: Disabled | | `sysctl` *[Sysctl](#sysctl) array* | (Optional) Sysctl configures sysctl parameters for tuning plugin | | `linuxPolicySetupTimeoutSeconds` *integer* | (Optional) LinuxPolicySetupTimeoutSeconds delays new pods from running containers until their policy has been programmed in the dataplane. The specified delay defines the maximum amount of time that the Calico CNI plugin will wait for policy to be programmed. Only applies to pods created on Linux nodes. \* A value of 0 disables pod startup delays. Default: 0 | ### CalicoNodeDaemonSet[​](#caliconodedaemonset) CalicoNodeDaemonSet is the configuration for the calico-node DaemonSet. *Appears in:* - [InstallationSpec](#installationspec) | Field | Description | | ------------------------------------------------------------ | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[CalicoNodeDaemonSetSpec](#caliconodedaemonsetspec)* | (Optional) Spec is the specification of the calico-node DaemonSet. | ### CalicoNodeDaemonSetContainer[​](#caliconodedaemonsetcontainer) CalicoNodeDaemonSetContainer is a calico-node DaemonSet container. *Appears in:* - [CalicoNodeDaemonSetPodSpec](#caliconodedaemonsetpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the calico-node DaemonSet container by name. Supported values are: calico-node | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named calico-node DaemonSet container's resources. If omitted, the calico-node DaemonSet will use its default value for this container's resources. If used in conjunction with the deprecated ComponentResources, then this value takes precedence. | ### CalicoNodeDaemonSetInitContainer[​](#caliconodedaemonsetinitcontainer) CalicoNodeDaemonSetInitContainer is a calico-node DaemonSet init container. *Appears in:* - [CalicoNodeDaemonSetPodSpec](#caliconodedaemonsetpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the calico-node DaemonSet init container by name. Supported values are: install-cni, hostpath-init, flexvol-driver, ebpf-bootstrap, node-certs-key-cert-provisioner, calico-node-prometheus-server-tls-key-cert-provisioner, mount-bpffs (deprecated, replaced by ebpf-bootstrap) | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named calico-node DaemonSet init container's resources. If omitted, the calico-node DaemonSet will use its default value for this container's resources. If used in conjunction with the deprecated ComponentResources, then this value takes precedence. | ### CalicoNodeDaemonSetPodSpec[​](#caliconodedaemonsetpodspec) CalicoNodeDaemonSetPodSpec is the calico-node DaemonSet's PodSpec. *Appears in:* - [CalicoNodeDaemonSetPodTemplateSpec](#caliconodedaemonsetpodtemplatespec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[CalicoNodeDaemonSetInitContainer](#caliconodedaemonsetinitcontainer) array* | (Optional) InitContainers is a list of calico-node init containers. If specified, this overrides the specified calico-node DaemonSet init containers. If omitted, the calico-node DaemonSet will use its default values for its init containers. | | `containers` *[CalicoNodeDaemonSetContainer](#caliconodedaemonsetcontainer) array* | (Optional) Containers is a list of calico-node containers. If specified, this overrides the specified calico-node DaemonSet containers. If omitted, the calico-node DaemonSet will use its default values for its containers. | | `affinity` *[Affinity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#affinity-v1-core)* | (Optional) Affinity is a group of affinity scheduling rules for the calico-node pods. If specified, this overrides any affinity that may be set on the calico-node DaemonSet. If omitted, the calico-node DaemonSet will use its default value for affinity. WARNING: Please note that this field will override the default calico-node DaemonSet affinity. | | `nodeSelector` *object (keys:string, values:string)* | (Optional) NodeSelector is the calico-node pod's scheduling constraints. If specified, each of the key/value pairs are added to the calico-node DaemonSet nodeSelector provided the key does not already exist in the object's nodeSelector. If omitted, the calico-node DaemonSet will use its default value for nodeSelector. WARNING: Please note that this field will modify the default calico-node DaemonSet nodeSelector. | | `tolerations` *[Toleration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#toleration-v1-core) array* | (Optional) Tolerations is the calico-node pod's tolerations. If specified, this overrides any tolerations that may be set on the calico-node DaemonSet. If omitted, the calico-node DaemonSet will use its default value for tolerations. WARNING: Please note that this field will override the default calico-node DaemonSet tolerations. | | `dnsPolicy` *[DNSPolicy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#dnspolicy-v1-core)* | (Optional) DNSPolicy is the DNS policy for the calico-node pods. | | `dnsConfig` *[PodDNSConfig](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#poddnsconfig-v1-core)* | (Optional) DNSConfig allows customization of the DNS configuration for the calico-node pods. | ### CalicoNodeDaemonSetPodTemplateSpec[​](#caliconodedaemonsetpodtemplatespec) CalicoNodeDaemonSetPodTemplateSpec is the calico-node DaemonSet's PodTemplateSpec *Appears in:* - [CalicoNodeDaemonSetSpec](#caliconodedaemonsetspec) | Field | Description | | ------------------------------------------------------------------ | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[CalicoNodeDaemonSetPodSpec](#caliconodedaemonsetpodspec)* | (Optional) Spec is the calico-node DaemonSet's PodSpec. | ### CalicoNodeDaemonSetSpec[​](#caliconodedaemonsetspec) CalicoNodeDaemonSetSpec defines configuration for the calico-node DaemonSet. *Appears in:* - [CalicoNodeDaemonSet](#caliconodedaemonset) | Field | Description | | -------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `minReadySeconds` *integer* | (Optional) MinReadySeconds is the minimum number of seconds for which a newly created DaemonSet pod should be ready without any of its container crashing, for it to be considered available. If specified, this overrides any minReadySeconds value that may be set on the calico-node DaemonSet. If omitted, the calico-node DaemonSet will use its default value for minReadySeconds. | | `template` *[CalicoNodeDaemonSetPodTemplateSpec](#caliconodedaemonsetpodtemplatespec)* | (Optional) Template describes the calico-node DaemonSet pod that will be created. | ### CalicoNodeWindowsDaemonSet[​](#caliconodewindowsdaemonset) CalicoNodeWindowsDaemonSet is the configuration for the calico-node-windows DaemonSet. *Appears in:* - [InstallationSpec](#installationspec) | Field | Description | | -------------------------------------------------------------------------- | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[CalicoNodeWindowsDaemonSetSpec](#caliconodewindowsdaemonsetspec)* | (Optional) Spec is the specification of the calico-node-windows DaemonSet. | ### CalicoNodeWindowsDaemonSetContainer[​](#caliconodewindowsdaemonsetcontainer) CalicoNodeWindowsDaemonSetContainer is a calico-node-windows DaemonSet container. *Appears in:* - [CalicoNodeWindowsDaemonSetPodSpec](#caliconodewindowsdaemonsetpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the calico-node-windows DaemonSet container by name. Supported values are: calico-node-windows | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named calico-node-windows DaemonSet container's resources. If omitted, the calico-node-windows DaemonSet will use its default value for this container's resources. If used in conjunction with the deprecated ComponentResources, then this value takes precedence. | ### CalicoNodeWindowsDaemonSetInitContainer[​](#caliconodewindowsdaemonsetinitcontainer) CalicoNodeWindowsDaemonSetInitContainer is a calico-node-windows DaemonSet init container. *Appears in:* - [CalicoNodeWindowsDaemonSetPodSpec](#caliconodewindowsdaemonsetpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the calico-node-windows DaemonSet init container by name. Supported values are: install-cni;hostpath-init, flexvol-driver, node-certs-key-cert-provisioner, calico-node-windows-prometheus-server-tls-key-cert-provisioner | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named calico-node-windows DaemonSet init container's resources. If omitted, the calico-node-windows DaemonSet will use its default value for this container's resources. If used in conjunction with the deprecated ComponentResources, then this value takes precedence. | ### CalicoNodeWindowsDaemonSetPodSpec[​](#caliconodewindowsdaemonsetpodspec) CalicoNodeWindowsDaemonSetPodSpec is the calico-node-windows DaemonSet's PodSpec. *Appears in:* - [CalicoNodeWindowsDaemonSetPodTemplateSpec](#caliconodewindowsdaemonsetpodtemplatespec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[CalicoNodeWindowsDaemonSetInitContainer](#caliconodewindowsdaemonsetinitcontainer) array* | (Optional) InitContainers is a list of calico-node-windows init containers. If specified, this overrides the specified calico-node-windows DaemonSet init containers. If omitted, the calico-node-windows DaemonSet will use its default values for its init containers. | | `containers` *[CalicoNodeWindowsDaemonSetContainer](#caliconodewindowsdaemonsetcontainer) array* | (Optional) Containers is a list of calico-node-windows containers. If specified, this overrides the specified calico-node-windows DaemonSet containers. If omitted, the calico-node-windows DaemonSet will use its default values for its containers. | | `affinity` *[Affinity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#affinity-v1-core)* | (Optional) Affinity is a group of affinity scheduling rules for the calico-node-windows pods. If specified, this overrides any affinity that may be set on the calico-node-windows DaemonSet. If omitted, the calico-node-windows DaemonSet will use its default value for affinity. WARNING: Please note that this field will override the default calico-node-windows DaemonSet affinity. | | `nodeSelector` *object (keys:string, values:string)* | (Optional) NodeSelector is the calico-node-windows pod's scheduling constraints. If specified, each of the key/value pairs are added to the calico-node-windows DaemonSet nodeSelector provided the key does not already exist in the object's nodeSelector. If omitted, the calico-node-windows DaemonSet will use its default value for nodeSelector. WARNING: Please note that this field will modify the default calico-node-windows DaemonSet nodeSelector. | | `tolerations` *[Toleration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#toleration-v1-core) array* | (Optional) Tolerations is the calico-node-windows pod's tolerations. If specified, this overrides any tolerations that may be set on the calico-node-windows DaemonSet. If omitted, the calico-node-windows DaemonSet will use its default value for tolerations. WARNING: Please note that this field will override the default calico-node-windows DaemonSet tolerations. | ### CalicoNodeWindowsDaemonSetPodTemplateSpec[​](#caliconodewindowsdaemonsetpodtemplatespec) CalicoNodeWindowsDaemonSetPodTemplateSpec is the calico-node-windows DaemonSet's PodTemplateSpec *Appears in:* - [CalicoNodeWindowsDaemonSetSpec](#caliconodewindowsdaemonsetspec) | Field | Description | | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[CalicoNodeWindowsDaemonSetPodSpec](#caliconodewindowsdaemonsetpodspec)* | (Optional) Spec is the calico-node-windows DaemonSet's PodSpec. | ### CalicoNodeWindowsDaemonSetSpec[​](#caliconodewindowsdaemonsetspec) CalicoNodeWindowsDaemonSetSpec defines configuration for the calico-node-windows DaemonSet. *Appears in:* - [CalicoNodeWindowsDaemonSet](#caliconodewindowsdaemonset) | Field | Description | | ---------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `minReadySeconds` *integer* | (Optional) MinReadySeconds is the minimum number of seconds for which a newly created DaemonSet pod should be ready without any of its container crashing, for it to be considered available. If specified, this overrides any minReadySeconds value that may be set on the calico-node-windows DaemonSet. If omitted, the calico-node-windows DaemonSet will use its default value for minReadySeconds. | | `template` *[CalicoNodeWindowsDaemonSetPodTemplateSpec](#caliconodewindowsdaemonsetpodtemplatespec)* | (Optional) Template describes the calico-node-windows DaemonSet pod that will be created. | ### CalicoWindowsUpgradeDaemonSet[​](#calicowindowsupgradedaemonset) Deprecated. The CalicoWindowsUpgradeDaemonSet is deprecated and will be removed from the API in the future. CalicoWindowsUpgradeDaemonSet is the configuration for the calico-windows-upgrade DaemonSet. *Appears in:* - [InstallationSpec](#installationspec) | Field | Description | | -------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[CalicoWindowsUpgradeDaemonSetSpec](#calicowindowsupgradedaemonsetspec)* | (Optional) Spec is the specification of the calico-windows-upgrade DaemonSet. | ### CalicoWindowsUpgradeDaemonSetContainer[​](#calicowindowsupgradedaemonsetcontainer) CalicoWindowsUpgradeDaemonSetContainer is a calico-windows-upgrade DaemonSet container. *Appears in:* - [CalicoWindowsUpgradeDaemonSetPodSpec](#calicowindowsupgradedaemonsetpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the calico-windows-upgrade DaemonSet container by name. | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named calico-windows-upgrade DaemonSet container's resources. If omitted, the calico-windows-upgrade DaemonSet will use its default value for this container's resources. | ### CalicoWindowsUpgradeDaemonSetPodSpec[​](#calicowindowsupgradedaemonsetpodspec) CalicoWindowsUpgradeDaemonSetPodSpec is the calico-windows-upgrade DaemonSet's PodSpec. *Appears in:* - [CalicoWindowsUpgradeDaemonSetPodTemplateSpec](#calicowindowsupgradedaemonsetpodtemplatespec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `containers` *[CalicoWindowsUpgradeDaemonSetContainer](#calicowindowsupgradedaemonsetcontainer) array* | (Optional) Containers is a list of calico-windows-upgrade containers. If specified, this overrides the specified calico-windows-upgrade DaemonSet containers. If omitted, the calico-windows-upgrade DaemonSet will use its default values for its containers. | | `affinity` *[Affinity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#affinity-v1-core)* | (Optional) Affinity is a group of affinity scheduling rules for the calico-windows-upgrade pods. If specified, this overrides any affinity that may be set on the calico-windows-upgrade DaemonSet. If omitted, the calico-windows-upgrade DaemonSet will use its default value for affinity. WARNING: Please note that this field will override the default calico-windows-upgrade DaemonSet affinity. | | `nodeSelector` *object (keys:string, values:string)* | (Optional) NodeSelector is the calico-windows-upgrade pod's scheduling constraints. If specified, each of the key/value pairs are added to the calico-windows-upgrade DaemonSet nodeSelector provided the key does not already exist in the object's nodeSelector. If omitted, the calico-windows-upgrade DaemonSet will use its default value for nodeSelector. WARNING: Please note that this field will modify the default calico-windows-upgrade DaemonSet nodeSelector. | | `tolerations` *[Toleration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#toleration-v1-core) array* | (Optional) Tolerations is the calico-windows-upgrade pod's tolerations. If specified, this overrides any tolerations that may be set on the calico-windows-upgrade DaemonSet. If omitted, the calico-windows-upgrade DaemonSet will use its default value for tolerations. WARNING: Please note that this field will override the default calico-windows-upgrade DaemonSet tolerations. | ### CalicoWindowsUpgradeDaemonSetPodTemplateSpec[​](#calicowindowsupgradedaemonsetpodtemplatespec) CalicoWindowsUpgradeDaemonSetPodTemplateSpec is the calico-windows-upgrade DaemonSet's PodTemplateSpec *Appears in:* - [CalicoWindowsUpgradeDaemonSetSpec](#calicowindowsupgradedaemonsetspec) | Field | Description | | -------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[CalicoWindowsUpgradeDaemonSetPodSpec](#calicowindowsupgradedaemonsetpodspec)* | (Optional) Spec is the calico-windows-upgrade DaemonSet's PodSpec. | ### CalicoWindowsUpgradeDaemonSetSpec[​](#calicowindowsupgradedaemonsetspec) CalicoWindowsUpgradeDaemonSetSpec defines configuration for the calico-windows-upgrade DaemonSet. *Appears in:* - [CalicoWindowsUpgradeDaemonSet](#calicowindowsupgradedaemonset) | Field | Description | | ---------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `minReadySeconds` *integer* | (Optional) MinReadySeconds is the minimum number of seconds for which a newly created Deployment pod should be ready without any of its container crashing, for it to be considered available. If specified, this overrides any minReadySeconds value that may be set on the calico-windows-upgrade DaemonSet. If omitted, the calico-windows-upgrade DaemonSet will use its default value for minReadySeconds. | | `template` *[CalicoWindowsUpgradeDaemonSetPodTemplateSpec](#calicowindowsupgradedaemonsetpodtemplatespec)* | (Optional) Template describes the calico-windows-upgrade DaemonSet pod that will be created. | ### CertificateManagement[​](#certificatemanagement) CertificateManagement configures pods to submit a CertificateSigningRequest to the certificates.k8s.io/v1beta1 API in order to obtain TLS certificates. This feature requires that you bring your own CSR signing and approval process, otherwise pods will be stuck during initialization. *Appears in:* - [InstallationSpec](#installationspec) | Field | Description | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `caCert` *integer array* | Certificate of the authority that signs the CertificateSigningRequests in PEM format. | | `signerName` *string* | When a CSR is issued to the certificates.k8s.io API, the signerName is added to the request in order to accommodate for clusters with multiple signers. Must be formatted as: `/`. | | `keyAlgorithm` *string* | (Optional) Specify the algorithm used by pods to generate a key pair that is associated with the X.509 certificate request. Default: RSAWithSize2048 | | `signatureAlgorithm` *string* | (Optional) Specify the algorithm used for the signature of the X.509 certificate request. Default: SHA256WithRSA | ### CollectProcessPathOption[​](#collectprocesspathoption) *Underlying type:* *string* *Appears in:* - [LogCollectorSpec](#logcollectorspec) | Value | Description | | ---------- | ----------- | | `Enabled` | | | `Disabled` | | ### CommonPrometheusFields[​](#commonprometheusfields) *Appears in:* - [PrometheusSpec](#prometheusspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `containers` *[PrometheusContainer](#prometheuscontainer) array* | (Optional) Containers is a list of Prometheus containers. If specified, this overrides the specified Prometheus Deployment containers. If omitted, the Prometheus Deployment will use its default values for its containers. | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | Define resources requests and limits for single Pods. | ### Compliance[​](#compliance) Compliance installs the components required for Tigera compliance reporting. At most one instance of this resource is supported. It must be named "tigera-secure". | Field | Description | | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------- | | `apiVersion` *string* | `operator.tigera.io/v1` | | `kind` *string* | `Compliance` | | `metadata` *[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#objectmeta-v1-meta)* | Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[ComplianceSpec](#compliancespec)* | Specification of the desired state for Tigera compliance reporting. | | `status` *[ComplianceStatus](#compliancestatus)* | Most recently observed state for Tigera compliance reporting. | ### ComplianceBenchmarkerDaemonSet[​](#compliancebenchmarkerdaemonset) ComplianceBenchmarkerDaemonSet is the configuration for the Compliance Benchmarker DaemonSet. *Appears in:* - [ComplianceSpec](#compliancespec) | Field | Description | | ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | | `spec` *[ComplianceBenchmarkerDaemonSetSpec](#compliancebenchmarkerdaemonsetspec)* | (Optional) Spec is the specification of the Compliance Benchmarker DaemonSet. | ### ComplianceBenchmarkerDaemonSetContainer[​](#compliancebenchmarkerdaemonsetcontainer) ComplianceBenchmarkerDaemonSetContainer is a Compliance Benchmarker DaemonSet container. *Appears in:* - [ComplianceBenchmarkerDaemonSetPodSpec](#compliancebenchmarkerdaemonsetpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the Compliance Benchmarker DaemonSet container by name. Supported values are: compliance-benchmarker | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named Compliance Benchmarker DaemonSet container's resources. If omitted, the Compliance Benchmarker DaemonSet will use its default value for this container's resources. | ### ComplianceBenchmarkerDaemonSetInitContainer[​](#compliancebenchmarkerdaemonsetinitcontainer) ComplianceBenchmarkerDaemonSetInitContainer is a Compliance Benchmarker DaemonSet init container. *Appears in:* - [ComplianceBenchmarkerDaemonSetPodSpec](#compliancebenchmarkerdaemonsetpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the Compliance Benchmarker DaemonSet init container by name. Supported values are: tigera-compliance-benchmarker-tls-key-cert-provisioner | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named Compliance Benchmarker DaemonSet init container's resources. If omitted, the Compliance Benchmarker DaemonSet will use its default value for this init container's resources. | ### ComplianceBenchmarkerDaemonSetPodSpec[​](#compliancebenchmarkerdaemonsetpodspec) ComplianceBenchmarkerDaemonSetPodSpec is the Compliance Benchmarker DaemonSet's PodSpec. *Appears in:* - [ComplianceBenchmarkerDaemonSetPodTemplateSpec](#compliancebenchmarkerdaemonsetpodtemplatespec) | Field | Description | | -------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[ComplianceBenchmarkerDaemonSetInitContainer](#compliancebenchmarkerdaemonsetinitcontainer) array* | (Optional) InitContainers is a list of Compliance benchmark init containers. If specified, this overrides the specified Compliance Benchmarker DaemonSet init containers. If omitted, the Compliance Benchmarker DaemonSet will use its default values for its init containers. | | `containers` *[ComplianceBenchmarkerDaemonSetContainer](#compliancebenchmarkerdaemonsetcontainer) array* | (Optional) Containers is a list of Compliance benchmark containers. If specified, this overrides the specified Compliance Benchmarker DaemonSet containers. If omitted, the Compliance Benchmarker DaemonSet will use its default values for its containers. | ### ComplianceBenchmarkerDaemonSetPodTemplateSpec[​](#compliancebenchmarkerdaemonsetpodtemplatespec) ComplianceBenchmarkerDaemonSetPodTemplateSpec is the Compliance Benchmarker DaemonSet's PodTemplateSpec *Appears in:* - [ComplianceBenchmarkerDaemonSetSpec](#compliancebenchmarkerdaemonsetspec) | Field | Description | | ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | | `spec` *[ComplianceBenchmarkerDaemonSetPodSpec](#compliancebenchmarkerdaemonsetpodspec)* | (Optional) Spec is the Compliance Benchmarker DaemonSet's PodSpec. | ### ComplianceBenchmarkerDaemonSetSpec[​](#compliancebenchmarkerdaemonsetspec) ComplianceBenchmarkerDaemonSetSpec defines configuration for the Compliance Benchmarker DaemonSet. *Appears in:* - [ComplianceBenchmarkerDaemonSet](#compliancebenchmarkerdaemonset) | Field | Description | | ------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------- | | `template` *[ComplianceBenchmarkerDaemonSetPodTemplateSpec](#compliancebenchmarkerdaemonsetpodtemplatespec)* | (Optional) Template describes the Compliance Benchmarker DaemonSet pod that will be created. | ### ComplianceControllerDeployment[​](#compliancecontrollerdeployment) ComplianceControllerDeployment is the configuration for the compliance controller Deployment. *Appears in:* - [ComplianceSpec](#compliancespec) | Field | Description | | ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | | `spec` *[ComplianceControllerDeploymentSpec](#compliancecontrollerdeploymentspec)* | (Optional) Spec is the specification of the compliance controller Deployment. | ### ComplianceControllerDeploymentContainer[​](#compliancecontrollerdeploymentcontainer) ComplianceControllerDeploymentContainer is a compliance controller Deployment container. *Appears in:* - [ComplianceControllerDeploymentPodSpec](#compliancecontrollerdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the compliance controller Deployment container by name. Supported values are: compliance-controller | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named compliance controller Deployment container's resources. If omitted, the compliance controller Deployment will use its default value for this container's resources. | ### ComplianceControllerDeploymentInitContainer[​](#compliancecontrollerdeploymentinitcontainer) ComplianceControllerDeploymentInitContainer is a compliance controller Deployment init container. *Appears in:* - [ComplianceControllerDeploymentPodSpec](#compliancecontrollerdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the compliance controller Deployment init container by name. Supported values are: tigera-compliance-controller-tls-key-cert-provisioner | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named compliance controller Deployment init container's resources. If omitted, the compliance controller Deployment will use its default value for this init container's resources. | ### ComplianceControllerDeploymentPodSpec[​](#compliancecontrollerdeploymentpodspec) ComplianceControllerDeploymentPodSpec is the compliance controller Deployment's PodSpec. *Appears in:* - [ComplianceControllerDeploymentPodTemplateSpec](#compliancecontrollerdeploymentpodtemplatespec) | Field | Description | | -------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[ComplianceControllerDeploymentInitContainer](#compliancecontrollerdeploymentinitcontainer) array* | (Optional) InitContainers is a list of compliance controller init containers. If specified, this overrides the specified compliance controller Deployment init containers. If omitted, the compliance controller Deployment will use its default values for its init containers. | | `containers` *[ComplianceControllerDeploymentContainer](#compliancecontrollerdeploymentcontainer) array* | (Optional) Containers is a list of compliance controller containers. If specified, this overrides the specified compliance controller Deployment containers. If omitted, the compliance controller Deployment will use its default values for its containers. | ### ComplianceControllerDeploymentPodTemplateSpec[​](#compliancecontrollerdeploymentpodtemplatespec) ComplianceControllerDeploymentPodTemplateSpec is the compliance controller Deployment's PodTemplateSpec *Appears in:* - [ComplianceControllerDeploymentSpec](#compliancecontrollerdeploymentspec) | Field | Description | | ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | | `spec` *[ComplianceControllerDeploymentPodSpec](#compliancecontrollerdeploymentpodspec)* | (Optional) Spec is the compliance controller Deployment's PodSpec. | ### ComplianceControllerDeploymentSpec[​](#compliancecontrollerdeploymentspec) ComplianceControllerDeploymentSpec defines configuration for the compliance controller Deployment. *Appears in:* - [ComplianceControllerDeployment](#compliancecontrollerdeployment) | Field | Description | | ------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------- | | `template` *[ComplianceControllerDeploymentPodTemplateSpec](#compliancecontrollerdeploymentpodtemplatespec)* | (Optional) Template describes the compliance controller Deployment pod that will be created. | ### ComplianceReporterPodSpec[​](#compliancereporterpodspec) ComplianceReporterPodSpec is the ComplianceReporter PodSpec. *Appears in:* - [ComplianceReporterPodTemplateSpec](#compliancereporterpodtemplatespec) | Field | Description | | ------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[ComplianceReporterPodTemplateInitContainer](#compliancereporterpodtemplateinitcontainer) array* | (Optional) InitContainers is a list of ComplianceReporter PodSpec init containers. If specified, this overrides the specified ComplianceReporter PodSpec init containers. If omitted, the ComplianceServer Deployment will use its default values for its init containers. | | `containers` *[ComplianceReporterPodTemplateContainer](#compliancereporterpodtemplatecontainer) array* | (Optional) Containers is a list of ComplianceServer containers. If specified, this overrides the specified ComplianceReporter PodSpec containers. If omitted, the ComplianceServer Deployment will use its default values for its containers. | ### ComplianceReporterPodTemplate[​](#compliancereporterpodtemplate) ComplianceReporterPodTemplate is the configuration for the ComplianceReporter PodTemplate. *Appears in:* - [ComplianceSpec](#compliancespec) | Field | Description | | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------- | | `template` *[ComplianceReporterPodTemplateSpec](#compliancereporterpodtemplatespec)* | (Optional) Spec is the specification of the ComplianceReporter PodTemplateSpec. | ### ComplianceReporterPodTemplateContainer[​](#compliancereporterpodtemplatecontainer) ComplianceReporterPodTemplateContainer is a ComplianceServer Deployment container. *Appears in:* - [ComplianceReporterPodSpec](#compliancereporterpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the ComplianceServer Deployment container by name. Supported values are: reporter | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named ComplianceServer Deployment container's resources. If omitted, the ComplianceServer Deployment will use its default value for this container's resources. | ### ComplianceReporterPodTemplateInitContainer[​](#compliancereporterpodtemplateinitcontainer) ComplianceReporterPodTemplateInitContainer is a ComplianceServer Deployment init container. *Appears in:* - [ComplianceReporterPodSpec](#compliancereporterpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `name` *string* | Name is an enum which identifies the ComplianceReporter PodSpec init container by name. Supported values are: tigera-compliance-reporter-tls-key-cert-provisioner | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named ComplianceReporter PodSpec init container's resources. If omitted, the ComplianceServer Deployment will use its default value for this init container's resources. | ### ComplianceReporterPodTemplateSpec[​](#compliancereporterpodtemplatespec) ComplianceReporterPodTemplateSpec is the ComplianceReporter PodTemplateSpec. *Appears in:* - [ComplianceReporterPodTemplate](#compliancereporterpodtemplate) | Field | Description | | ---------------------------------------------------------------- | ---------------------------------------------------------------- | | `spec` *[ComplianceReporterPodSpec](#compliancereporterpodspec)* | (Optional) Spec is the ComplianceReporter PodTemplate's PodSpec. | ### ComplianceServerDeployment[​](#complianceserverdeployment) ComplianceServerDeployment is the configuration for the ComplianceServer Deployment. *Appears in:* - [ComplianceSpec](#compliancespec) | Field | Description | | -------------------------------------------------------------------------- | ------------------------------------------------------------------------ | | `spec` *[ComplianceServerDeploymentSpec](#complianceserverdeploymentspec)* | (Optional) Spec is the specification of the ComplianceServer Deployment. | ### ComplianceServerDeploymentContainer[​](#complianceserverdeploymentcontainer) ComplianceServerDeploymentContainer is a ComplianceServer Deployment container. *Appears in:* - [ComplianceServerDeploymentPodSpec](#complianceserverdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the ComplianceServer Deployment container by name. Supported values are: compliance-server | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named ComplianceServer Deployment container's resources. If omitted, the ComplianceServer Deployment will use its default value for this container's resources. | ### ComplianceServerDeploymentInitContainer[​](#complianceserverdeploymentinitcontainer) ComplianceServerDeploymentInitContainer is a ComplianceServer Deployment init container. *Appears in:* - [ComplianceServerDeploymentPodSpec](#complianceserverdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the ComplianceServer Deployment init container by name. Supported values are: tigera-compliance-server-tls-key-cert-provisioner | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named ComplianceServer Deployment init container's resources. If omitted, the ComplianceServer Deployment will use its default value for this init container's resources. | ### ComplianceServerDeploymentPodSpec[​](#complianceserverdeploymentpodspec) ComplianceServerDeploymentPodSpec is the ComplianceServer Deployment's PodSpec. *Appears in:* - [ComplianceServerDeploymentPodTemplateSpec](#complianceserverdeploymentpodtemplatespec) | Field | Description | | ------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[ComplianceServerDeploymentInitContainer](#complianceserverdeploymentinitcontainer) array* | (Optional) InitContainers is a list of ComplianceServer init containers. If specified, this overrides the specified ComplianceServer Deployment init containers. If omitted, the ComplianceServer Deployment will use its default values for its init containers. | | `containers` *[ComplianceServerDeploymentContainer](#complianceserverdeploymentcontainer) array* | (Optional) Containers is a list of ComplianceServer containers. If specified, this overrides the specified ComplianceServer Deployment containers. If omitted, the ComplianceServer Deployment will use its default values for its containers. | ### ComplianceServerDeploymentPodTemplateSpec[​](#complianceserverdeploymentpodtemplatespec) ComplianceServerDeploymentPodTemplateSpec is the ComplianceServer Deployment's PodTemplateSpec *Appears in:* - [ComplianceServerDeploymentSpec](#complianceserverdeploymentspec) | Field | Description | | -------------------------------------------------------------------------------- | ------------------------------------------------------------- | | `spec` *[ComplianceServerDeploymentPodSpec](#complianceserverdeploymentpodspec)* | (Optional) Spec is the ComplianceServer Deployment's PodSpec. | ### ComplianceServerDeploymentSpec[​](#complianceserverdeploymentspec) ComplianceServerDeploymentSpec defines configuration for the ComplianceServer Deployment. *Appears in:* - [ComplianceServerDeployment](#complianceserverdeployment) | Field | Description | | ---------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | `template` *[ComplianceServerDeploymentPodTemplateSpec](#complianceserverdeploymentpodtemplatespec)* | (Optional) Template describes the ComplianceServer Deployment pod that will be created. | ### ComplianceSnapshotterDeployment[​](#compliancesnapshotterdeployment) ComplianceSnapshotterDeployment is the configuration for the compliance snapshotter Deployment. *Appears in:* - [ComplianceSpec](#compliancespec) | Field | Description | | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------ | | `spec` *[ComplianceSnapshotterDeploymentSpec](#compliancesnapshotterdeploymentspec)* | (Optional) Spec is the specification of the compliance snapshotter Deployment. | ### ComplianceSnapshotterDeploymentContainer[​](#compliancesnapshotterdeploymentcontainer) ComplianceSnapshotterDeploymentContainer is a compliance snapshotter Deployment container. *Appears in:* - [ComplianceSnapshotterDeploymentPodSpec](#compliancesnapshotterdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the compliance snapshotter Deployment container by name. Supported values are: compliance-snapshotter | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named compliance snapshotter Deployment container's resources. If omitted, the compliance snapshotter Deployment will use its default value for this container's resources. | ### ComplianceSnapshotterDeploymentInitContainer[​](#compliancesnapshotterdeploymentinitcontainer) ComplianceSnapshotterDeploymentInitContainer is a compliance snapshotter Deployment init container. *Appears in:* - [ComplianceSnapshotterDeploymentPodSpec](#compliancesnapshotterdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the compliance snapshotter Deployment init container by name. Supported values are: tigera-compliance-snapshotter-tls-key-cert-provisioner | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named compliance snapshotter Deployment init container's resources. If omitted, the compliance snapshotter Deployment will use its default value for this init container's resources. | ### ComplianceSnapshotterDeploymentPodSpec[​](#compliancesnapshotterdeploymentpodspec) ComplianceSnapshotterDeploymentPodSpec is the compliance snapshotter Deployment's PodSpec. *Appears in:* - [ComplianceSnapshotterDeploymentPodTemplateSpec](#compliancesnapshotterdeploymentpodtemplatespec) | Field | Description | | ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[ComplianceSnapshotterDeploymentInitContainer](#compliancesnapshotterdeploymentinitcontainer) array* | (Optional) InitContainers is a list of compliance snapshotter init containers. If specified, this overrides the specified compliance snapshotter Deployment init containers. If omitted, the compliance snapshotter Deployment will use its default values for its init containers. | | `containers` *[ComplianceSnapshotterDeploymentContainer](#compliancesnapshotterdeploymentcontainer) array* | (Optional) Containers is a list of compliance snapshotter containers. If specified, this overrides the specified compliance snapshotter Deployment containers. If omitted, the compliance snapshotter Deployment will use its default values for its containers. | ### ComplianceSnapshotterDeploymentPodTemplateSpec[​](#compliancesnapshotterdeploymentpodtemplatespec) ComplianceSnapshotterDeploymentPodTemplateSpec is the compliance snapshotter Deployment's PodTemplateSpec *Appears in:* - [ComplianceSnapshotterDeploymentSpec](#compliancesnapshotterdeploymentspec) | Field | Description | | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------- | | `spec` *[ComplianceSnapshotterDeploymentPodSpec](#compliancesnapshotterdeploymentpodspec)* | (Optional) Spec is the compliance snapshotter Deployment's PodSpec. | ### ComplianceSnapshotterDeploymentSpec[​](#compliancesnapshotterdeploymentspec) ComplianceSnapshotterDeploymentSpec defines configuration for the compliance snapshotter Deployment. *Appears in:* - [ComplianceSnapshotterDeployment](#compliancesnapshotterdeployment) | Field | Description | | -------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | | `template` *[ComplianceSnapshotterDeploymentPodTemplateSpec](#compliancesnapshotterdeploymentpodtemplatespec)* | (Optional) Template describes the compliance snapshotter Deployment pod that will be created. | ### ComplianceSpec[​](#compliancespec) ComplianceSpec defines the desired state of Tigera compliance reporting capabilities. *Appears in:* - [Compliance](#compliance) | Field | Description | | ------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | | `complianceControllerDeployment` *[ComplianceControllerDeployment](#compliancecontrollerdeployment)* | (Optional) ComplianceControllerDeployment configures the Compliance Controller Deployment. | | `complianceSnapshotterDeployment` *[ComplianceSnapshotterDeployment](#compliancesnapshotterdeployment)* | (Optional) ComplianceSnapshotterDeployment configures the Compliance Snapshotter Deployment. | | `complianceBenchmarkerDaemonSet` *[ComplianceBenchmarkerDaemonSet](#compliancebenchmarkerdaemonset)* | (Optional) ComplianceBenchmarkerDaemonSet configures the Compliance Benchmarker DaemonSet. | | `complianceServerDeployment` *[ComplianceServerDeployment](#complianceserverdeployment)* | (Optional) ComplianceServerDeployment configures the Compliance Server Deployment. | | `complianceReporterPodTemplate` *[ComplianceReporterPodTemplate](#compliancereporterpodtemplate)* | (Optional) ComplianceReporterPodTemplate configures the Compliance Reporter PodTemplate. | ### ComplianceStatus[​](#compliancestatus) ComplianceStatus defines the observed state of Tigera compliance reporting capabilities. *Appears in:* - [Compliance](#compliance) | Field | Description | | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `state` *string* | State provides user-readable status. | | `conditions` *[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#condition-v1-meta) array* | (Optional) Conditions represents the latest observed set of conditions for the component. A component may be one or more of Ready, Progressing, Degraded or other customer types. | ### ComponentName[​](#componentname) *Underlying type:* *string* ComponentName represents a single component. One of: Node, Typha, KubeControllers *Appears in:* - [ComponentResource](#componentresource) | Value | Description | | ----------------- | ----------- | | `Node` | | | `NodeWindows` | | | `FelixWindows` | | | `ConfdWindows` | | | `Typha` | | | `KubeControllers` | | ### ComponentResource[​](#componentresource) Deprecated. Please use component resource config fields in Installation.Spec instead. The ComponentResource struct associates a ResourceRequirements with a component by name *Appears in:* - [InstallationSpec](#installationspec) | Field | Description | | -------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | `componentName` *[ComponentName](#componentname)* | ComponentName is an enum which identifies the component | | `resourceRequirements` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | ResourceRequirements allows customization of limits and requests for compute resources such as cpu and memory. | ### ConditionStatus[​](#conditionstatus) *Underlying type:* *string* ConditionStatus represents the status of a particular condition. A condition may be one of: True, False, Unknown. *Appears in:* - [TigeraStatusCondition](#tigerastatuscondition) | Value | Description | | --------- | ----------- | | `True` | | | `False` | | | `Unknown` | | ### ContainerIPForwardingType[​](#containeripforwardingtype) *Underlying type:* *string* ContainerIPForwardingType specifies whether the CNI config for container ip forwarding is enabled. *Appears in:* - [CalicoNetworkSpec](#caliconetworkspec) | Value | Description | | ---------- | ----------- | | `Enabled` | | | `Disabled` | | ### DPIDaemonsetInitContainer[​](#dpidaemonsetinitcontainer) *Appears in:* - [DPIDaemonsetTemplateSpec](#dpidaemonsettemplatespec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum that identifies the init container by its name. | | `image` *string* | Image name for the init container | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the init container's resources. If omitted, the default values will be used for the init container's resources. | ### DPIDaemonsetSpec[​](#dpidaemonsetspec) *Appears in:* - [DeepPacketInspectionDaemonset](#deeppacketinspectiondaemonset) | Field | Description | | ---------------------------------------------------------- | ---------------------------------------------------- | | `template` *[DPIDaemonsetTemplate](#dpidaemonsettemplate)* | (Optional) Template specifies DPI Daemonset Template | ### DPIDaemonsetTemplate[​](#dpidaemonsettemplate) *Appears in:* - [DPIDaemonsetSpec](#dpidaemonsetspec) | Field | Description | | -------------------------------------------------------------- | ----------------------------------------------------- | | `spec` *[DPIDaemonsetTemplateSpec](#dpidaemonsettemplatespec)* | (Optional) Spec specifies DPI Daemonset Template Spec | ### DPIDaemonsetTemplateSpec[​](#dpidaemonsettemplatespec) *Appears in:* - [DPIDaemonsetTemplate](#dpidaemonsettemplate) | Field | Description | | -------------------------------------------------------------------------------- | ------------------------------------------------- | | `initContainers` *[DPIDaemonsetInitContainer](#dpidaemonsetinitcontainer) array* | List of DPI Daemonset Init containers definitions | ### DashboardsJobContainer[​](#dashboardsjobcontainer) DashboardsJobContainer is the Dashboards job container. *Appears in:* - [DashboardsJobPodSpec](#dashboardsjobpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the Dashboard Job container by name. Supported values are: dashboards-installer | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named Dashboard Job container's resources. If omitted, the Dashboard Job will use its default value for this container's resources. | ### DashboardsJobPodSpec[​](#dashboardsjobpodspec) DashboardsJobPodSpec is the Dashboards job's PodSpec. *Appears in:* - [DashboardsJobPodTemplateSpec](#dashboardsjobpodtemplatespec) | Field | Description | | ---------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `containers` *[DashboardsJobContainer](#dashboardsjobcontainer) array* | (Optional) Containers is a list of dashboards job containers. If specified, this overrides the specified Dashboard job containers. If omitted, the Dashboard job will use its default values for its containers. | ### DashboardsJobPodTemplateSpec[​](#dashboardsjobpodtemplatespec) DashboardsJobPodTemplateSpec is the Dashboards job's PodTemplateSpec *Appears in:* - [DashboardsJobSpec](#dashboardsjobspec) | Field | Description | | ------------------------------------------------------ | ----------------------------------------------- | | `spec` *[DashboardsJobPodSpec](#dashboardsjobpodspec)* | (Optional) Spec is the Dashboard job's PodSpec. | ### DashboardsJobSpec[​](#dashboardsjobspec) DashboardsJobSpec defines configuration for the Dashboards job. *Appears in:* - [DashboardsJob](#dashboardsjob) | Field | Description | | -------------------------------------------------------------------------- | -------------------------------------------------------------------------- | | `template` *[DashboardsJobPodTemplateSpec](#dashboardsjobpodtemplatespec)* | (Optional) Template describes the Dashboards job pod that will be created. | ### DataType[​](#datatype) *Underlying type:* *string* DataType represent the type of data stored *Validation:* - Enum: \[Alerts AuditLogs BGPLogs ComplianceBenchmarks ComplianceReports ComplianceSnapshots DNSLogs FlowLogs L7Logs RuntimeReports ThreatFeedsDomainSet ThreatFeedsIPSet WAFLogs] *Appears in:* - [Index](#index) | Value | Description | | ---------------------- | ----------- | | `Alerts` | | | `AuditLogs` | | | `BGPLogs` | | | `ComplianceBenchmarks` | | | `ComplianceReports` | | | `ComplianceSnapshots` | | | `DNSLogs` | | | `FlowLogs` | | | `L7Logs` | | | `RuntimeReports` | | | `ThreatFeedsDomainSet` | | | `ThreatFeedsIPSet` | | | `WAFLogs` | | ### DeepPacketInspectionDaemonset[​](#deeppacketinspectiondaemonset) *Appears in:* - [IntrusionDetectionSpec](#intrusiondetectionspec) | Field | Description | | ---------------------------------------------- | -------------------------------------------------------- | | `spec` *[DPIDaemonsetSpec](#dpidaemonsetspec)* | (Optional) DPIDaemonsetSpec configures the DPI Daemonset | ### DexDeployment[​](#dexdeployment) DexDeployment is the configuration for the Dex Deployment. *Appears in:* - [AuthenticationSpec](#authenticationspec) | Field | Description | | ------------------------------------------------ | ----------------------------------------------------------- | | `spec` *[DexDeploymentSpec](#dexdeploymentspec)* | (Optional) Spec is the specification of the Dex Deployment. | ### DexDeploymentContainer[​](#dexdeploymentcontainer) DexDeploymentContainer is a Dex Deployment container. *Appears in:* - [DexDeploymentPodSpec](#dexdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the Dex Deployment container by name. Supported values are: tigera-dex | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named Dex Deployment container's resources. If omitted, the Dex Deployment will use its default value for this container's resources. | ### DexDeploymentInitContainer[​](#dexdeploymentinitcontainer) DexDeploymentInitContainer is a Dex Deployment init container. *Appears in:* - [DexDeploymentPodSpec](#dexdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the Dex Deployment init container by name. Supported values are: tigera-dex-tls-key-cert-provisioner | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named Dex Deployment init container's resources. If omitted, the Dex Deployment will use its default value for this init container's resources. | ### DexDeploymentPodSpec[​](#dexdeploymentpodspec) DexDeploymentPodSpec is the Dex Deployment's PodSpec. *Appears in:* - [DexDeploymentPodTemplateSpec](#dexdeploymentpodtemplatespec) | Field | Description | | ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[DexDeploymentInitContainer](#dexdeploymentinitcontainer) array* | (Optional) InitContainers is a list of Dex init containers. If specified, this overrides the specified Dex Deployment init containers. If omitted, the Dex Deployment will use its default values for its init containers. | | `containers` *[DexDeploymentContainer](#dexdeploymentcontainer) array* | (Optional) Containers is a list of Dex containers. If specified, this overrides the specified Dex Deployment containers. If omitted, the Dex Deployment will use its default values for its containers. | ### DexDeploymentPodTemplateSpec[​](#dexdeploymentpodtemplatespec) DexDeploymentPodTemplateSpec is the Dex Deployment's PodTemplateSpec *Appears in:* - [DexDeploymentSpec](#dexdeploymentspec) | Field | Description | | ------------------------------------------------------ | ------------------------------------------------ | | `spec` *[DexDeploymentPodSpec](#dexdeploymentpodspec)* | (Optional) Spec is the Dex Deployment's PodSpec. | ### DexDeploymentSpec[​](#dexdeploymentspec) DexDeploymentSpec defines configuration for the Dex Deployment. *Appears in:* - [DexDeployment](#dexdeployment) | Field | Description | | -------------------------------------------------------------------------- | -------------------------------------------------------------------------- | | `template` *[DexDeploymentPodTemplateSpec](#dexdeploymentpodtemplatespec)* | (Optional) Template describes the Dex Deployment pod that will be created. | ### ECKOperatorStatefulSet[​](#eckoperatorstatefulset) ECKOperatorStatefulSet is the configuration for the ECKOperator StatefulSet. *Appears in:* - [LogStorageSpec](#logstoragespec) | Field | Description | | ------------------------------------------------------------------ | -------------------------------------------------------------------- | | `spec` *[ECKOperatorStatefulSetSpec](#eckoperatorstatefulsetspec)* | (Optional) Spec is the specification of the ECKOperator StatefulSet. | ### ECKOperatorStatefulSetContainer[​](#eckoperatorstatefulsetcontainer) ECKOperatorStatefulSetContainer is a ECKOperator StatefulSet container. *Appears in:* - [ECKOperatorStatefulSetPodSpec](#eckoperatorstatefulsetpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the ECKOperator StatefulSet container by name. Supported values are: manager | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named ECKOperator StatefulSet container's resources. If omitted, the ECKOperator StatefulSet will use its default value for this container's resources. | ### ECKOperatorStatefulSetInitContainer[​](#eckoperatorstatefulsetinitcontainer) ECKOperatorStatefulSetInitContainer is a ECKOperator StatefulSet init container. *Appears in:* - [ECKOperatorStatefulSetPodSpec](#eckoperatorstatefulsetpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the ECKOperator StatefulSet init container by name. | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named ECKOperator StatefulSet init container's resources. If omitted, the ECKOperator StatefulSet will use its default value for this init container's resources. | ### ECKOperatorStatefulSetPodSpec[​](#eckoperatorstatefulsetpodspec) ECKOperatorStatefulSetPodSpec is the ECKOperator StatefulSet's PodSpec. *Appears in:* - [ECKOperatorStatefulSetPodTemplateSpec](#eckoperatorstatefulsetpodtemplatespec) | Field | Description | | ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[ECKOperatorStatefulSetInitContainer](#eckoperatorstatefulsetinitcontainer) array* | (Optional) InitContainers is a list of ECKOperator StatefulSet init containers. If specified, this overrides the specified ECKOperator StatefulSet init containers. If omitted, the ECKOperator StatefulSet will use its default values for its init containers. | | `containers` *[ECKOperatorStatefulSetContainer](#eckoperatorstatefulsetcontainer) array* | (Optional) Containers is a list of ECKOperator StatefulSet containers. If specified, this overrides the specified ECKOperator StatefulSet containers. If omitted, the ECKOperator StatefulSet will use its default values for its containers. | ### ECKOperatorStatefulSetPodTemplateSpec[​](#eckoperatorstatefulsetpodtemplatespec) ECKOperatorStatefulSetPodTemplateSpec is the ECKOperator StatefulSet's PodTemplateSpec *Appears in:* - [ECKOperatorStatefulSetSpec](#eckoperatorstatefulsetspec) | Field | Description | | ------------------------------------------------------------------------ | --------------------------------------------------------- | | `spec` *[ECKOperatorStatefulSetPodSpec](#eckoperatorstatefulsetpodspec)* | (Optional) Spec is the ECKOperator StatefulSet's PodSpec. | ### ECKOperatorStatefulSetSpec[​](#eckoperatorstatefulsetspec) ECKOperatorStatefulSetSpec defines configuration for the ECKOperator StatefulSet. *Appears in:* - [ECKOperatorStatefulSet](#eckoperatorstatefulset) | Field | Description | | -------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | | `template` *[ECKOperatorStatefulSetPodTemplateSpec](#eckoperatorstatefulsetpodtemplatespec)* | (Optional) Template describes the ECKOperator StatefulSet pod that will be created. | ### EGWDeploymentContainer[​](#egwdeploymentcontainer) EGWDeploymentContainer is a Egress Gateway Deployment container. *Appears in:* - [EgressGatewayDeploymentPodSpec](#egressgatewaydeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the EGW Deployment container by name. Supported values are: calico-egw | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named EGW Deployment container's resources. If omitted, the EGW Deployment will use its default value for this container's resources. If used in conjunction with the deprecated ComponentResources, then this value takes precedence. | ### EGWDeploymentInitContainer[​](#egwdeploymentinitcontainer) EGWDeploymentInitContainer is a Egress Gateway Deployment init container. *Appears in:* - [EgressGatewayDeploymentPodSpec](#egressgatewaydeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `name` *string* | Name is an enum which identifies the EGW Deployment init container by name. Supported values are: egress-gateway-init | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named EGW Deployment init container's resources. If omitted, the EGW Deployment will use its default value for this init container's resources. If used in conjunction with the deprecated ComponentResources, then this value takes precedence. | ### EKSLogForwarderDeployment[​](#ekslogforwarderdeployment) EKSLogForwarderDeployment is the configuration for the EKSLogForwarder Deployment. *Appears in:* - [LogCollectorSpec](#logcollectorspec) | Field | Description | | ------------------------------------------------------------------------ | ----------------------------------------------------------------------- | | `spec` *[EKSLogForwarderDeploymentSpec](#ekslogforwarderdeploymentspec)* | (Optional) Spec is the specification of the EKSLogForwarder Deployment. | ### EKSLogForwarderDeploymentContainer[​](#ekslogforwarderdeploymentcontainer) EKSLogForwarderDeploymentContainer is a EKSLogForwarder Deployment container. *Appears in:* - [EKSLogForwarderDeploymentPodSpec](#ekslogforwarderdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the EKSLogForwarder Deployment container by name. Supported values are: eks-log-forwarder | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named EKSLogForwarder Deployment container's resources. If omitted, the EKSLogForwarder Deployment will use its default value for this container's resources. | ### EKSLogForwarderDeploymentInitContainer[​](#ekslogforwarderdeploymentinitcontainer) EKSLogForwarderDeploymentInitContainer is a EKSLogForwarder Deployment init container. *Appears in:* - [EKSLogForwarderDeploymentPodSpec](#ekslogforwarderdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the EKSLogForwarder Deployment init container by name. Supported values are: eks-log-forwarder-startup | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named EKSLogForwarder Deployment init container's resources. If omitted, the EKSLogForwarder Deployment will use its default value for this init container's resources. | ### EKSLogForwarderDeploymentPodSpec[​](#ekslogforwarderdeploymentpodspec) EKSLogForwarderDeploymentPodSpec is the EKSLogForwarder Deployment's PodSpec. *Appears in:* - [EKSLogForwarderDeploymentPodTemplateSpec](#ekslogforwarderdeploymentpodtemplatespec) | Field | Description | | ---------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[EKSLogForwarderDeploymentInitContainer](#ekslogforwarderdeploymentinitcontainer) array* | (Optional) InitContainers is a list of EKSLogForwarder init containers. If specified, this overrides the specified EKSLogForwarder Deployment init containers. If omitted, the EKSLogForwarder Deployment will use its default values for its init containers. | | `containers` *[EKSLogForwarderDeploymentContainer](#ekslogforwarderdeploymentcontainer) array* | (Optional) Containers is a list of EKSLogForwarder containers. If specified, this overrides the specified EKSLogForwarder Deployment containers. If omitted, the EKSLogForwarder Deployment will use its default values for its containers. | ### EKSLogForwarderDeploymentPodTemplateSpec[​](#ekslogforwarderdeploymentpodtemplatespec) EKSLogForwarderDeploymentPodTemplateSpec is the EKSLogForwarder Deployment's PodTemplateSpec *Appears in:* - [EKSLogForwarderDeploymentSpec](#ekslogforwarderdeploymentspec) | Field | Description | | ------------------------------------------------------------------------------ | ------------------------------------------------------------ | | `spec` *[EKSLogForwarderDeploymentPodSpec](#ekslogforwarderdeploymentpodspec)* | (Optional) Spec is the EKSLogForwarder Deployment's PodSpec. | ### EKSLogForwarderDeploymentSpec[​](#ekslogforwarderdeploymentspec) EKSLogForwarderDeploymentSpec defines configuration for the EKSLogForwarder Deployment. *Appears in:* - [EKSLogForwarderDeployment](#ekslogforwarderdeployment) | Field | Description | | -------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | | `template` *[EKSLogForwarderDeploymentPodTemplateSpec](#ekslogforwarderdeploymentpodtemplatespec)* | (Optional) Template describes the EKSLogForwarder Deployment pod that will be created. | ### ESGatewayDeployment[​](#esgatewaydeployment) ESGatewayDeployment is the configuration for the es-gateway Deployment. *Appears in:* - [LogStorageSpec](#logstoragespec) | Field | Description | | ------------------------------------------------------------ | ------------------------------------------------------------------ | | `spec` *[ESGatewayDeploymentSpec](#esgatewaydeploymentspec)* | (Optional) Spec is the specification of the es-gateway Deployment. | ### ESGatewayDeploymentContainer[​](#esgatewaydeploymentcontainer) ESGatewayDeploymentContainer is a es-gateway Deployment container. *Appears in:* - [ESGatewayDeploymentPodSpec](#esgatewaydeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the es-gateway Deployment container by name. Supported values are: tigera-secure-es-gateway | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named es-gateway Deployment container's resources. If omitted, the es-gateway Deployment will use its default value for this container's resources. | ### ESGatewayDeploymentInitContainer[​](#esgatewaydeploymentinitcontainer) ESGatewayDeploymentInitContainer is a es-gateway Deployment init container. *Appears in:* - [ESGatewayDeploymentPodSpec](#esgatewaydeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the es-gateway Deployment init container by name. Supported values are: tigera-secure-elasticsearch-cert-key-cert-provisioner | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named es-gateway Deployment init container's resources. If omitted, the es-gateway Deployment will use its default value for this init container's resources. | ### ESGatewayDeploymentPodSpec[​](#esgatewaydeploymentpodspec) ESGatewayDeploymentPodSpec is the es-gateway Deployment's PodSpec. *Appears in:* - [ESGatewayDeploymentPodTemplateSpec](#esgatewaydeploymentpodtemplatespec) | Field | Description | | ---------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[ESGatewayDeploymentInitContainer](#esgatewaydeploymentinitcontainer) array* | (Optional) InitContainers is a list of es-gateway init containers. If specified, this overrides the specified es-gateway Deployment init containers. If omitted, the es-gateway Deployment will use its default values for its init containers. | | `containers` *[ESGatewayDeploymentContainer](#esgatewaydeploymentcontainer) array* | (Optional) Containers is a list of es-gateway containers. If specified, this overrides the specified es-gateway Deployment containers. If omitted, the es-gateway Deployment will use its default values for its containers. | ### ESGatewayDeploymentPodTemplateSpec[​](#esgatewaydeploymentpodtemplatespec) ESGatewayDeploymentPodTemplateSpec is the es-gateway Deployment's PodTemplateSpec *Appears in:* - [ESGatewayDeploymentSpec](#esgatewaydeploymentspec) | Field | Description | | ------------------------------------------------------------------ | ------------------------------------------------------- | | `spec` *[ESGatewayDeploymentPodSpec](#esgatewaydeploymentpodspec)* | (Optional) Spec is the es-gateway Deployment's PodSpec. | ### ESGatewayDeploymentSpec[​](#esgatewaydeploymentspec) ESGatewayDeploymentSpec defines configuration for the es-gateway Deployment. *Appears in:* - [ESGatewayDeployment](#esgatewaydeployment) | Field | Description | | -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | | `template` *[ESGatewayDeploymentPodTemplateSpec](#esgatewaydeploymentpodtemplatespec)* | (Optional) Template describes the es-gateway Deployment pod that will be created. | ### EgressGateway[​](#egressgateway) EgressGateway is the Schema for the egressgateways API | Field | Description | | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------- | | `apiVersion` *string* | `operator.tigera.io/v1` | | `kind` *string* | `EgressGateway` | | `metadata` *[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#objectmeta-v1-meta)* | Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[EgressGatewaySpec](#egressgatewayspec)* | | | `status` *[EgressGatewayStatus](#egressgatewaystatus)* | | ### EgressGatewayDeploymentPodSpec[​](#egressgatewaydeploymentpodspec) EgressGatewayDeploymentPodSpec is the Egress Gateway Deployment's PodSpec. *Appears in:* - [EgressGatewayDeploymentPodTemplateSpec](#egressgatewaydeploymentpodtemplatespec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `initContainers` *[EGWDeploymentInitContainer](#egwdeploymentinitcontainer) array* | (Optional) InitContainers is a list of EGW init containers. If specified, this overrides the specified EGW Deployment init containers. If omitted, the EGW Deployment will use its default values for its init containers. | | `containers` *[EGWDeploymentContainer](#egwdeploymentcontainer) array* | (Optional) Containers is a list of EGW containers. If specified, this overrides the specified EGW Deployment containers. If omitted, the EGW Deployment will use its default values for its containers. | | `affinity` *[Affinity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#affinity-v1-core)* | (Optional) Affinity is a group of affinity scheduling rules for the EGW pods. | | `nodeSelector` *object (keys:string, values:string)* | (Optional) NodeSelector gives more control over the nodes where the Egress Gateway pods will run on. | | `terminationGracePeriodSeconds` *integer* | (Optional) TerminationGracePeriodSeconds defines the termination grace period of the Egress Gateway pods in seconds. | | `topologySpreadConstraints` *[TopologySpreadConstraint](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#topologyspreadconstraint-v1-core) array* | (Optional) TopologySpreadConstraints defines how the Egress Gateway pods should be spread across different AZs. | | `tolerations` *[Toleration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#toleration-v1-core) array* | (Optional) Tolerations is the egress gateway pod's tolerations. If specified, this overrides any tolerations that may be set on the EGW Deployment. If omitted, the EGW Deployment will use its default value for tolerations. | | `priorityClassName` *string* | (Optional) PriorityClassName allows to specify a PriorityClass resource to be used. | ### EgressGatewayDeploymentPodTemplateSpec[​](#egressgatewaydeploymentpodtemplatespec) EgressGatewayDeploymentPodTemplateSpec is the EGW Deployment's PodTemplateSpec *Appears in:* - [EgressGatewaySpec](#egressgatewayspec) | Field | Description | | -------------------------------------------------------------------------- | -------------------------------------------------------------------------- | | `metadata` *[EgressGatewayMetadata](#egressgatewaymetadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[EgressGatewayDeploymentPodSpec](#egressgatewaydeploymentpodspec)* | (Optional) Spec is the EGW Deployment's PodSpec. | ### EgressGatewayFailureDetection[​](#egressgatewayfailuredetection) EgressGatewayFailureDetection defines the fields the needed for determining Egress Gateway readiness. *Appears in:* - [EgressGatewaySpec](#egressgatewayspec) | Field | Description | | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `healthTimeoutDataStoreSeconds` *integer* | (Optional) HealthTimeoutDataStoreSeconds defines how long Egress Gateway can fail to connect to the datastore before reporting not ready. This value must be greater than 0. Default: 90 | | `icmpProbe` *[ICMPProbe](#icmpprobe)* | (Optional) ICMPProbe define outgoing ICMP probes that Egress Gateway will use to verify its upstream connection. Egress Gateway will report not ready if all fail. Timeout must be greater than interval. | | `httpProbe` *[HTTPProbe](#httpprobe)* | (Optional) HTTPProbe define outgoing HTTP probes that Egress Gateway will use to verify its upsteam connection. Egress Gateway will report not ready if all fail. Timeout must be greater than interval. | ### EgressGatewayIPPool[​](#egressgatewayippool) *Appears in:* - [EgressGatewaySpec](#egressgatewayspec) | Field | Description | | --------------- | --------------------------------------------------------------------------- | | `name` *string* | (Optional) Name is the name of the IPPool that the Egress Gateways can use. | | `cidr` *string* | (Optional) CIDR is the IPPool CIDR that the Egress Gateways can use. | ### EgressGatewayMetadata[​](#egressgatewaymetadata) EgressGatewayMetadata contains the standard Kubernetes labels and annotations fields. *Appears in:* - [EgressGatewayDeploymentPodTemplateSpec](#egressgatewaydeploymentpodtemplatespec) | Field | Description | | ----------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `labels` *object (keys:string, values:string)* | (Optional) Labels is a map of string keys and values that may match replica set and service selectors. Each of these key/value pairs are added to the object's labels provided the key does not already exist in the object's labels. If not specified will default to projectcalico.org/egw:\[name], where \[name] is the name of the Egress Gateway resource. | | `annotations` *object (keys:string, values:string)* | (Optional) Annotations is a map of arbitrary non-identifying metadata. Each of these key/value pairs are added to the object's annotations provided the key does not already exist in the object's annotations. | ### EgressGatewaySpec[​](#egressgatewayspec) EgressGatewaySpec defines the desired state of EgressGateway *Appears in:* - [EgressGateway](#egressgateway) | Field | Description | | ------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `replicas` *integer* | (Optional) Replicas defines how many instances of the Egress Gateway pod will run. | | `ipPools` *[EgressGatewayIPPool](#egressgatewayippool) array* | IPPools defines the IP Pools that the Egress Gateway pods should be using. Either name or CIDR must be specified. IPPools must match existing IPPools. | | `externalNetworks` *string array* | (Optional) ExternalNetworks defines the external network names this Egress Gateway is associated with. ExternalNetworks must match existing external networks. | | `logSeverity` *[LogSeverity](#logseverity)* | (Optional) LogSeverity defines the logging level of the Egress Gateway. | | `template` *[EgressGatewayDeploymentPodTemplateSpec](#egressgatewaydeploymentpodtemplatespec)* | (Optional) Template describes the EGW Deployment pod that will be created. | | `egressGatewayFailureDetection` *[EgressGatewayFailureDetection](#egressgatewayfailuredetection)* | (Optional) EgressGatewayFailureDetection is used to configure how Egress Gateway determines readiness. If both ICMP, HTTP probes are defined, one ICMP probe and one HTTP probe should succeed for Egress Gateways to become ready. Otherwise one of ICMP or HTTP probe should succeed for Egress gateways to become ready if configured. | | `aws` *[AWSEgressGateway](#awsegressgateway)* | (Optional) AWS defines the additional configuration options for Egress Gateways on AWS. | ### EgressGatewayStatus[​](#egressgatewaystatus) EgressGatewayStatus defines the observed state of EgressGateway *Appears in:* - [EgressGateway](#egressgateway) | Field | Description | | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `state` *string* | State provides user-readable status. | | `conditions` *[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#condition-v1-meta) array* | (Optional) Conditions represents the latest observed set of conditions for the component. A component may be one or more of Ready, Progressing, Degraded or other customer types. | ### EksCloudwatchLogsSpec[​](#ekscloudwatchlogsspec) EksConfigSpec defines configuration for fetching EKS audit logs. *Appears in:* - [AdditionalLogSourceSpec](#additionallogsourcespec) | Field | Description | | ------------------------- | --------------------------------------------------------------------------------------------------------------------- | | `region` *string* | AWS Region EKS cluster is hosted in. | | `groupName` *string* | Cloudwatch log-group name containing EKS audit logs. | | `streamPrefix` *string* | (Optional) Prefix of Cloudwatch log stream containing EKS audit logs in the log-group. Default: kube-apiserver-audit- | | `fetchInterval` *integer* | (Optional) Cloudwatch audit logs fetching interval in seconds. Default: 60 | ### ElasticsearchMetricsDeployment[​](#elasticsearchmetricsdeployment) ElasticsearchMetricsDeployment is the configuration for the tigera-elasticsearch-metric Deployment. *Appears in:* - [LogStorageSpec](#logstoragespec) | Field | Description | | ---------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | | `spec` *[ElasticsearchMetricsDeploymentSpec](#elasticsearchmetricsdeploymentspec)* | (Optional) Spec is the specification of the ElasticsearchMetrics Deployment. | ### ElasticsearchMetricsDeploymentContainer[​](#elasticsearchmetricsdeploymentcontainer) ElasticsearchMetricsDeploymentContainer is a ElasticsearchMetricsDeployment container. *Appears in:* - [ElasticsearchMetricsDeploymentPodSpec](#elasticsearchmetricsdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the ElasticsearchMetricsDeployment container by name. Supported values are: tigera-elasticsearch-metrics | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named ElasticsearchMetricsDeployment container's resources. If omitted, the ElasticsearchMetrics Deployment will use its default value for this container's resources. | ### ElasticsearchMetricsDeploymentInitContainer[​](#elasticsearchmetricsdeploymentinitcontainer) ElasticsearchMetricsDeploymentInitContainer is a ElasticsearchMetricsDeployment init container. *Appears in:* - [ElasticsearchMetricsDeploymentPodSpec](#elasticsearchmetricsdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the ElasticsearchMetricsDeployment init container by name. Supported values are: tigera-ee-elasticsearch-metrics-tls-key-cert-provisioner | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named ElasticsearchMetricsDeployment init container's resources. If omitted, the ElasticsearchMetrics Deployment will use its default value for this init container's resources. | ### ElasticsearchMetricsDeploymentPodSpec[​](#elasticsearchmetricsdeploymentpodspec) ElasticsearchMetricsDeploymentPodSpec is the tElasticsearchMetricsDeployment's PodSpec. *Appears in:* - [ElasticsearchMetricsDeploymentPodTemplateSpec](#elasticsearchmetricsdeploymentpodtemplatespec) | Field | Description | | -------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[ElasticsearchMetricsDeploymentInitContainer](#elasticsearchmetricsdeploymentinitcontainer) array* | (Optional) InitContainers is a list of ElasticsearchMetricsDeployment init containers. If specified, this overrides the specified ElasticsearchMetricsDeployment init containers. If omitted, the ElasticsearchMetrics Deployment will use its default values for its init containers. | | `containers` *[ElasticsearchMetricsDeploymentContainer](#elasticsearchmetricsdeploymentcontainer) array* | (Optional) Containers is a list of ElasticsearchMetricsDeployment containers. If specified, this overrides the specified ElasticsearchMetricsDeployment containers. If omitted, the ElasticsearchMetrics Deployment will use its default values for its containers. | ### ElasticsearchMetricsDeploymentPodTemplateSpec[​](#elasticsearchmetricsdeploymentpodtemplatespec) ElasticsearchMetricsDeploymentPodTemplateSpec is the ElasticsearchMetricsDeployment's PodTemplateSpec *Appears in:* - [ElasticsearchMetricsDeploymentSpec](#elasticsearchmetricsdeploymentspec) | Field | Description | | ---------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | | `spec` *[ElasticsearchMetricsDeploymentPodSpec](#elasticsearchmetricsdeploymentpodspec)* | (Optional) Spec is the ElasticsearchMetrics Deployment's PodSpec. | ### ElasticsearchMetricsDeploymentSpec[​](#elasticsearchmetricsdeploymentspec) ElasticsearchMetricsDeploymentSpec defines configuration for the ElasticsearchMetricsDeployment Deployment. *Appears in:* - [ElasticsearchMetricsDeployment](#elasticsearchmetricsdeployment) | Field | Description | | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------- | | `template` *[ElasticsearchMetricsDeploymentPodTemplateSpec](#elasticsearchmetricsdeploymentpodtemplatespec)* | (Optional) Template describes the ElasticsearchMetrics Deployment pod that will be created. | ### EmailVerificationType[​](#emailverificationtype) *Underlying type:* *string* *Appears in:* - [AuthenticationOIDC](#authenticationoidc) | Value | Description | | -------------- | ----------- | | `Verify` | | | `InsecureSkip` | | ### EncapsulationType[​](#encapsulationtype) *Underlying type:* *string* EncapsulationType is the type of encapsulation to use on an IP pool. One of: IPIP, VXLAN, IPIPCrossSubnet, VXLANCrossSubnet, None *Appears in:* - [IPPool](#ippool) | Value | Description | | ------------------ | ----------- | | `IPIPCrossSubnet` | | | `IPIP` | | | `VXLAN` | | | `VXLANCrossSubnet` | | | `None` | | ### EncryptionOption[​](#encryptionoption) *Underlying type:* *string* EncryptionOption specifies the traffic encryption mode when connecting to a Syslog server. One of: None, TLS *Appears in:* - [SyslogStoreSpec](#syslogstorespec) | Value | Description | | ------ | ----------- | | `None` | | | `TLS` | | ### Endpoint[​](#endpoint) Endpoint contains a subset of relevant fields from the Prometheus Endpoint struct. *Appears in:* - [ServiceMonitor](#servicemonitor) | Field | Description | | ----------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `params` *object (keys:string, values:string array)* | Optional HTTP URL parameters Default: scrape all metrics. | | `bearerTokenSecret` *[SecretKeySelector](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#secretkeyselector-v1-core)* | Secret to mount to read bearer token for scraping targets. Recommended: when unset, the operator will create a Secret, a ClusterRole and a ClusterRoleBinding. | | `interval` *[Duration](#duration)* | Interval at which metrics should be scraped. If not specified Prometheus' global scrape interval is used. | | `scrapeTimeout` *[Duration](#duration)* | Timeout after which the scrape is ended. If not specified, the Prometheus global scrape timeout is used unless it is less than `Interval` in which the latter is used. | | `honorLabels` *boolean* | HonorLabels chooses the metric's labels on collisions with target labels. | | `honorTimestamps` *boolean* | HonorTimestamps controls whether Prometheus respects the timestamps present in scraped data. | | `metricRelabelings` *RelabelConfig array* | MetricRelabelConfigs to apply to samples before ingestion. | | `relabelings` *RelabelConfig array* | RelabelConfigs to apply to samples before scraping. Prometheus Operator automatically adds relabelings for a few standard Kubernetes fields. The original scrape job's name is available via the `__tmp_prometheus_job_name` label. More info: [https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel\_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) | ### EnvoySettings[​](#envoysettings) *Appears in:* - [ApplicationLayerSpec](#applicationlayerspec) | Field | Description | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `xffNumTrustedHops` *integer* | (Optional) The number of additional ingress proxy hops from the right side of the x-forwarded-for HTTP header to trust when determining the origin client’s IP address. 0 is permitted, but >=1 is the typical setting. | | `useRemoteAddress` *boolean* | (Optional) If set to true, the Envoy connection manager will use the real remote address of the client connection when determining internal versus external origin and manipulating various headers. | ### ExternalPrometheus[​](#externalprometheus) *Appears in:* - [MonitorSpec](#monitorspec) | Field | Description | | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `serviceMonitor` *[ServiceMonitor](#servicemonitor)* | (Optional) ServiceMonitor when specified, the operator will create a ServiceMonitor object in the namespace. It is recommended that you configure labels if you want your prometheus instance to pick up the configuration automatically. The operator will configure 1 endpoint by default: - Params to scrape all metrics available in Calico Enterprise. - BearerTokenSecret (If not overridden, the operator will also create corresponding RBAC that allows authz to the metrics.) - TLSConfig, containing the caFile and serverName. | | `namespace` *string* | Namespace is the namespace where the operator will create resources for your Prometheus instance. The namespace must be created before the operator will create Prometheus resources. | ### FIPSMode[​](#fipsmode) *Underlying type:* *string* *Appears in:* - [InstallationSpec](#installationspec) | Value | Description | | ---------- | ----------- | | `Enabled` | | | `Disabled` | | ### FluentdDaemonSet[​](#fluentddaemonset) FluentdDaemonSet is the configuration for the Fluentd DaemonSet. *Appears in:* - [LogCollectorSpec](#logcollectorspec) | Field | Description | | ------------------------------------------------------ | -------------------------------------------------------------- | | `spec` *[FluentdDaemonSetSpec](#fluentddaemonsetspec)* | (Optional) Spec is the specification of the Fluentd DaemonSet. | ### FluentdDaemonSetContainer[​](#fluentddaemonsetcontainer) FluentdDaemonSetContainer is a Fluentd DaemonSet container. *Appears in:* - [FluentdDaemonSetPodSpec](#fluentddaemonsetpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the Fluentd DaemonSet container by name. Supported values are: fluentd | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named Fluentd DaemonSet container's resources. If omitted, the Fluentd DaemonSet will use its default value for this container's resources. | ### FluentdDaemonSetInitContainer[​](#fluentddaemonsetinitcontainer) FluentdDaemonSetInitContainer is a Fluentd DaemonSet init container. *Appears in:* - [FluentdDaemonSetPodSpec](#fluentddaemonsetpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the Fluentd DaemonSet init container by name. Supported values are: tigera-fluentd-prometheus-tls-key-cert-provisioner | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named Fluentd DaemonSet init container's resources. If omitted, the Fluentd DaemonSet will use its default value for this init container's resources. | ### FluentdDaemonSetPodSpec[​](#fluentddaemonsetpodspec) FluentdDaemonSetPodSpec is the Fluentd DaemonSet's PodSpec. *Appears in:* - [FluentdDaemonSetPodTemplateSpec](#fluentddaemonsetpodtemplatespec) | Field | Description | | ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[FluentdDaemonSetInitContainer](#fluentddaemonsetinitcontainer) array* | (Optional) InitContainers is a list of Fluentd DaemonSet init containers. If specified, this overrides the specified Fluentd DaemonSet init containers. If omitted, the Fluentd DaemonSet will use its default values for its init containers. | | `containers` *[FluentdDaemonSetContainer](#fluentddaemonsetcontainer) array* | (Optional) Containers is a list of Fluentd DaemonSet containers. If specified, this overrides the specified Fluentd DaemonSet containers. If omitted, the Fluentd DaemonSet will use its default values for its containers. | ### FluentdDaemonSetPodTemplateSpec[​](#fluentddaemonsetpodtemplatespec) FluentdDaemonSetPodTemplateSpec is the Fluentd DaemonSet's PodTemplateSpec *Appears in:* - [FluentdDaemonSetSpec](#fluentddaemonsetspec) | Field | Description | | ------------------------------------------------------------ | --------------------------------------------------- | | `spec` *[FluentdDaemonSetPodSpec](#fluentddaemonsetpodspec)* | (Optional) Spec is the Fluentd DaemonSet's PodSpec. | ### FluentdDaemonSetSpec[​](#fluentddaemonsetspec) FluentdDaemonSetSpec defines configuration for the Fluentd DaemonSet. *Appears in:* - [FluentdDaemonSet](#fluentddaemonset) | Field | Description | | -------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | | `template` *[FluentdDaemonSetPodTemplateSpec](#fluentddaemonsetpodtemplatespec)* | (Optional) Template describes the Fluentd DaemonSet pod that will be created. | ### GatewayAPI[​](#gatewayapi) | Field | Description | | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------- | | `apiVersion` *string* | `operator.tigera.io/v1` | | `kind` *string* | `GatewayAPI` | | `metadata` *[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#objectmeta-v1-meta)* | Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[GatewayAPISpec](#gatewayapispec)* | | ### GatewayAPISpec[​](#gatewayapispec) GatewayAPISpec has fields that can be used to customize our GatewayAPI support. *Appears in:* - [GatewayAPI](#gatewayapi) | Field | Description | | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `envoyGatewayConfigRef` *[NamespacedName](#namespacedname)* | (Optional) Reference to a custom EnvoyGateway YAML to use as the base EnvoyGateway configuration for the gateway controller. When specified, must identify a ConfigMap resource with an "envoy-gateway.yaml" key whose value is the desired EnvoyGateway YAML (i.e. following the same pattern as the default `envoy-gateway-config` ConfigMap). When not specified, the Tigera operator uses the `envoy-gateway-config` from the Envoy Gateway helm chart as its base. Starting from that base, the Tigera operator copies and modifies the EnvoyGateway resource as follows: 1. If not already specified, it sets the ControllerName to "gateway.envoyproxy.io/gatewayclass-controller". 2. It configures the `tigera/envoy-gateway` and `tigera/envoy-ratelimit` images that will be used (according to the current Calico version, private registry and image set settings) and any pull secrets that are needed to pull those images. 3. It enables use of the Backend API. The resulting EnvoyGateway is provisioned as the `envoy-gateway-config` ConfigMap (which the gateway controller then uses as its config). | | `gatewayClasses` *[GatewayClassSpec](#gatewayclassspec) array* | (Optional) Configures the GatewayClasses that will be available; please see GatewayClassSpec for more detail. If GatewayClasses is nil, the Tigera operator defaults to provisioning a single GatewayClass named "tigera-gateway-class", without any of the detailed customizations that are allowed within GatewayClassSpec. | | `gatewayControllerDeployment` *[GatewayControllerDeployment](#gatewaycontrollerdeployment)* | (Optional) Allows customization of the gateway controller deployment. | | `gatewayCertgenJob` *[GatewayCertgenJob](#gatewaycertgenjob)* | (Optional) Allows customization of the gateway certgen job. | | `crdManagement` *[CRDManagement](#crdmanagement)* | (Optional) Configures how to manage and update Gateway API CRDs. The default behaviour - which is used when this field is not set, or is set to "PreferExisting" - is that the Tigera operator will create the Gateway API CRDs if they do not already exist, but will not overwrite any existing Gateway API CRDs. This setting may be preferable if the customer is using other implementations of the Gateway API concurrently with the Gateway API support in Calico Enterprise. It is then the customer's responsibility to ensure that CRDs are installed that meet the needs of all the Gateway API implementations in their cluster. Alternatively, if this field is set to "Reconcile", the Tigera operator will keep the cluster's Gateway API CRDs aligned with those that it would install on a cluster that does not yet have any version of those CRDs. | ### GatewayCertgenJob[​](#gatewaycertgenjob) GatewayCertgenJob allows customization of the gateway certgen job. *Appears in:* - [GatewayAPISpec](#gatewayapispec) | Field | Description | | -------------------------------------------------------- | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[GatewayCertgenJobSpec](#gatewaycertgenjobspec)* | (Optional) | ### GatewayCertgenJobContainer[​](#gatewaycertgenjobcontainer) GatewayCertgenJobContainer allows customization of the gateway certgen job's resource requirements. *Appears in:* - [GatewayCertgenJobPodSpec](#gatewaycertgenjobpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | `name` *string* | | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) If non-nil, Resources sets the ResourceRequirements of the job's "envoy-gateway-certgen" container. | ### GatewayCertgenJobPodSpec[​](#gatewaycertgenjobpodspec) GatewayCertgenJobPodSpec allows customization of the gateway certgen job's pod spec. *Appears in:* - [GatewayCertgenJobPodTemplate](#gatewaycertgenjobpodtemplate) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | `affinity` *[Affinity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#affinity-v1-core)* | (Optional) If non-nil, Affinity sets the affinity field of the job's pod template. | | `containers` *[GatewayCertgenJobContainer](#gatewaycertgenjobcontainer) array* | (Optional) | | `nodeSelector` *object (keys:string, values:string)* | (Optional) If non-nil, NodeSelector sets the node selector for where job pods may be scheduled. | | `tolerations` *[Toleration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#toleration-v1-core) array* | (Optional) If non-nil, Tolerations sets the tolerations field of the job's pod template. | ### GatewayCertgenJobPodTemplate[​](#gatewaycertgenjobpodtemplate) GatewayCertgenJobPodTemplate allows customization of the gateway certgen job's pod template. *Appears in:* - [GatewayCertgenJobSpec](#gatewaycertgenjobspec) | Field | Description | | -------------------------------------------------------------- | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[GatewayCertgenJobPodSpec](#gatewaycertgenjobpodspec)* | (Optional) | ### GatewayCertgenJobSpec[​](#gatewaycertgenjobspec) GatewayCertgenJobSpec allows customization of the gateway certgen job spec. *Appears in:* - [GatewayCertgenJob](#gatewaycertgenjob) | Field | Description | | -------------------------------------------------------------------------- | ----------- | | `template` *[GatewayCertgenJobPodTemplate](#gatewaycertgenjobpodtemplate)* | (Optional) | ### GatewayClassSpec[​](#gatewayclassspec) *Appears in:* - [GatewayAPISpec](#gatewayapispec) | Field | Description | | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | The name of this GatewayClass. | | `envoyProxyRef` *[NamespacedName](#namespacedname)* | (Optional) Reference to a custom EnvoyProxy resource to use as the base EnvoyProxy configuration for this GatewayClass. When specified, must identify an EnvoyProxy resource. When not specified, the Tigera operator uses an empty EnvoyProxy resource as its base. Starting from that base, the Tigera operator copies and modifies the EnvoyProxy resource as follows, in the order described: 1. It configures the `tigera/envoy-proxy` image that will be used (according to the current Calico version, private registry and image set settings) and any pull secrets that are needed to pull that image. 2. It applies customizations as specified by the following `GatewayKind`, `GatewayDeployment`, `GatewayDaemonSet` and `GatewayService` fields. The resulting EnvoyProxy is provisioned in the `tigera-gateway` namespace, together with a GatewayClass that references it. If a custom EnvoyProxy resource is specified and uses `EnvoyDaemonSet` instead of the default `EnvoyDeployment`, deployment-related customizations will be applied within `EnvoyDaemonSet` instead of within `EnvoyDeployment`. | | `gatewayKind` *[GatewayKind](#gatewaykind)* | (Optional) Specifies whether Gateways in this class are deployed as Deployments (default) or as DaemonSets. It is an error for GatewayKind to specify a choice that is incompatible with the custom EnvoyProxy, when EnvoyProxyRef is also specified. | | `gatewayDeployment` *[GatewayDeployment](#gatewaydeployment)* | (Optional) Allows customization of Gateways when deployed as Kubernetes Deployments, for Gateways in this GatewayClass. | | `gatewayDaemonSet` *[GatewayDaemonSet](#gatewaydaemonset)* | (Optional) Allows customization of Gateways when deployed as Kubernetes DaemonSets, for Gateways in this GatewayClass. | | `gatewayService` *[GatewayService](#gatewayservice)* | (Optional) Allows customization of gateway services, for Gateways in this GatewayClass. | ### GatewayControllerDeployment[​](#gatewaycontrollerdeployment) GatewayControllerDeployment allows customization of the gateway controller deployment. *Appears in:* - [GatewayAPISpec](#gatewayapispec) | Field | Description | | ---------------------------------------------------------------------------- | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[GatewayControllerDeploymentSpec](#gatewaycontrollerdeploymentspec)* | (Optional) | ### GatewayControllerDeploymentContainer[​](#gatewaycontrollerdeploymentcontainer) GatewayControllerDeploymentContainer allows customization of the gateway controller's resource requirements. *Appears in:* - [GatewayControllerDeploymentPodSpec](#gatewaycontrollerdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | | `name` *string* | | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) If non-nil, Resources sets the ResourceRequirements of the controller's "envoy-gateway" container. | ### GatewayControllerDeploymentPodSpec[​](#gatewaycontrollerdeploymentpodspec) GatewayControllerDeploymentPodSpec allows customization of the gateway controller deployment pod spec. *Appears in:* - [GatewayControllerDeploymentPodTemplate](#gatewaycontrollerdeploymentpodtemplate) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `affinity` *[Affinity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#affinity-v1-core)* | (Optional) If non-nil, Affinity sets the affinity field of the deployment's pod template. | | `containers` *[GatewayControllerDeploymentContainer](#gatewaycontrollerdeploymentcontainer) array* | (Optional) | | `nodeSelector` *object (keys:string, values:string)* | (Optional) If non-nil, NodeSelector sets the node selector for where deployment pods may be scheduled. | | `topologySpreadConstraints` *[TopologySpreadConstraint](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#topologyspreadconstraint-v1-core) array* | (Optional) If non-nil, TopologySpreadConstraints sets the topology spread constraints of the deployment's pod template. TopologySpreadConstraints describes how a group of pods ought to spread across topology domains. Scheduler will schedule pods in a way which abides by the constraints. All topologySpreadConstraints are ANDed. | | `tolerations` *[Toleration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#toleration-v1-core) array* | (Optional) If non-nil, Tolerations sets the tolerations field of the deployment's pod template. | ### GatewayControllerDeploymentPodTemplate[​](#gatewaycontrollerdeploymentpodtemplate) GatewayControllerDeploymentPodTemplate allows customization of the gateway controller deployment pod template. *Appears in:* - [GatewayControllerDeploymentSpec](#gatewaycontrollerdeploymentspec) | Field | Description | | ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[GatewayControllerDeploymentPodSpec](#gatewaycontrollerdeploymentpodspec)* | (Optional) | ### GatewayControllerDeploymentSpec[​](#gatewaycontrollerdeploymentspec) GatewayControllerDeploymentSpec allows customization of the gateway controller deployment spec. *Appears in:* - [GatewayControllerDeployment](#gatewaycontrollerdeployment) | Field | Description | | ---------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | `replicas` *integer* | (Optional) If non-nil, Replicas sets the number of replicas for the deployment. | | `minReadySeconds` *integer* | (Optional) If non-nil, MinReadySeconds sets the minReadySeconds field for the deployment. | | `template` *[GatewayControllerDeploymentPodTemplate](#gatewaycontrollerdeploymentpodtemplate)* | (Optional) | ### GatewayDaemonSet[​](#gatewaydaemonset) GatewayDeployment allows customization of Gateways when deployed as Kubernetes DaemonSets. *Appears in:* - [GatewayClassSpec](#gatewayclassspec) | Field | Description | | ------------------------------------------------------ | ----------- | | `spec` *[GatewayDaemonSetSpec](#gatewaydaemonsetspec)* | (Optional) | ### GatewayDaemonSetContainer[​](#gatewaydaemonsetcontainer) GatewayDaemonSetContainer allows customization of the resource requirements of gateway daemonsets. *Appears in:* - [GatewayDaemonSetPodSpec](#gatewaydaemonsetpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | | `name` *string* | | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) If non-nil, Resources sets the ResourceRequirements of the daemonset's "envoy" container. | ### GatewayDaemonSetPodSpec[​](#gatewaydaemonsetpodspec) GatewayDaemonSetPodSpec allows customization of the pod spec of gateway daemonsets. *Appears in:* - [GatewayDaemonSetPodTemplate](#gatewaydaemonsetpodtemplate) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `affinity` *[Affinity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#affinity-v1-core)* | (Optional) If non-nil, Affinity sets the affinity field of the daemonset's pod template. | | `containers` *[GatewayDaemonSetContainer](#gatewaydaemonsetcontainer) array* | (Optional) | | `nodeSelector` *object (keys:string, values:string)* | (Optional) If non-nil, NodeSelector sets the node selector for where daemonset pods may be scheduled. | | `topologySpreadConstraints` *[TopologySpreadConstraint](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#topologyspreadconstraint-v1-core) array* | (Optional) If non-nil, TopologySpreadConstraints sets the topology spread constraints of the daemonset's pod template. TopologySpreadConstraints describes how a group of pods ought to spread across topology domains. Scheduler will schedule pods in a way which abides by the constraints. All topologySpreadConstraints are ANDed. | | `tolerations` *[Toleration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#toleration-v1-core) array* | (Optional) If non-nil, Tolerations sets the tolerations field of the daemonset's pod template. | ### GatewayDaemonSetPodTemplate[​](#gatewaydaemonsetpodtemplate) GatewayDeploymentPodTemplate allows customization of the pod template of gateway daemonsets. *Appears in:* - [GatewayDaemonSetSpec](#gatewaydaemonsetspec) | Field | Description | | ------------------------------------------------------------ | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[GatewayDaemonSetPodSpec](#gatewaydaemonsetpodspec)* | (Optional) | ### GatewayDaemonSetSpec[​](#gatewaydaemonsetspec) GatewayDeploymentSpec allows customization of the spec of gateway daemonsets. *Appears in:* - [GatewayDaemonSet](#gatewaydaemonset) | Field | Description | | ------------------------------------------------------------------------ | ----------- | | `template` *[GatewayDaemonSetPodTemplate](#gatewaydaemonsetpodtemplate)* | (Optional) | ### GatewayDeployment[​](#gatewaydeployment) GatewayDeployment allows customization of Gateways when deployed as Kubernetes Deployments. *Appears in:* - [GatewayClassSpec](#gatewayclassspec) | Field | Description | | -------------------------------------------------------- | ----------- | | `spec` *[GatewayDeploymentSpec](#gatewaydeploymentspec)* | (Optional) | ### GatewayDeploymentContainer[​](#gatewaydeploymentcontainer) GatewayDeploymentContainer allows customization of the resource requirements of gateway deployments. *Appears in:* - [GatewayDeploymentPodSpec](#gatewaydeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | | `name` *string* | | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) If non-nil, Resources sets the ResourceRequirements of the deployment's "envoy" container. | ### GatewayDeploymentPodSpec[​](#gatewaydeploymentpodspec) GatewayDeploymentPodSpec allows customization of the pod spec of gateway deployments. *Appears in:* - [GatewayDeploymentPodTemplate](#gatewaydeploymentpodtemplate) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `affinity` *[Affinity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#affinity-v1-core)* | (Optional) If non-nil, Affinity sets the affinity field of the deployment's pod template. | | `containers` *[GatewayDeploymentContainer](#gatewaydeploymentcontainer) array* | (Optional) | | `nodeSelector` *object (keys:string, values:string)* | (Optional) If non-nil, NodeSelector sets the node selector for where deployment pods may be scheduled. | | `topologySpreadConstraints` *[TopologySpreadConstraint](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#topologyspreadconstraint-v1-core) array* | (Optional) If non-nil, TopologySpreadConstraints sets the topology spread constraints of the deployment's pod template. TopologySpreadConstraints describes how a group of pods ought to spread across topology domains. Scheduler will schedule pods in a way which abides by the constraints. All topologySpreadConstraints are ANDed. | | `tolerations` *[Toleration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#toleration-v1-core) array* | (Optional) If non-nil, Tolerations sets the tolerations field of the deployment's pod template. | ### GatewayDeploymentPodTemplate[​](#gatewaydeploymentpodtemplate) GatewayDeploymentPodTemplate allows customization of the pod template of gateway deployments. *Appears in:* - [GatewayDeploymentSpec](#gatewaydeploymentspec) | Field | Description | | -------------------------------------------------------------- | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[GatewayDeploymentPodSpec](#gatewaydeploymentpodspec)* | (Optional) | ### GatewayDeploymentSpec[​](#gatewaydeploymentspec) GatewayDeploymentSpec allows customization of the spec of gateway deployments. *Appears in:* - [GatewayDeployment](#gatewaydeployment) | Field | Description | | -------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | | `replicas` *integer* | (Optional) If non-nil, Replicas sets the number of replicas for the deployment. | | `template` *[GatewayDeploymentPodTemplate](#gatewaydeploymentpodtemplate)* | (Optional) | | `strategy` *[GatewayDeploymentStrategy](#gatewaydeploymentstrategy)* | (Optional) The deployment strategy to use to replace existing pods with new ones. | ### GatewayDeploymentStrategy[​](#gatewaydeploymentstrategy) GatewayDeploymentStrategy allows customization of the deployment strategy for gateway deployments. If GatewayDeployment.Spec.Strategy is non-nil, gateway deployments are set to use a rolling update strategy, with the parameters specified in GatewayDeployment.Spec.Strategy. Only RollingUpdate is supported at this time so the Type field is not exposed. *Appears in:* - [GatewayDeploymentSpec](#gatewaydeploymentspec) | Field | Description | | ------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | | `rollingUpdate` *[RollingUpdateDeployment](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#rollingupdatedeployment-v1-apps)* | (Optional) | ### GatewayKind[​](#gatewaykind) *Underlying type:* *string* *Validation:* - Enum: \[Deployment DaemonSet] *Appears in:* - [GatewayClassSpec](#gatewayclassspec) | Value | Description | | ------------ | ----------- | | `Deployment` | | | `DaemonSet` | | ### GatewayService[​](#gatewayservice) GatewayService allows customization of the Services that front Gateways. *Appears in:* - [GatewayClassSpec](#gatewayclassspec) | Field | Description | | -------------------------------------------------- | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[GatewayServiceSpec](#gatewayservicespec)* | (Optional) | ### GatewayServiceSpec[​](#gatewayservicespec) GatewayServiceSpec allows customization of the services that front gateway deployments. The LoadBalancer fields allow customization of the corresponding fields in the Kubernetes ServiceSpec. These can be used for some cloud-independent control of the external load balancer that is provisioned for each Gateway. For finer-grained cloud-specific control please use the Metadata.Annotations field in GatewayService. *Appears in:* - [GatewayService](#gatewayservice) | Field | Description | | ----------------------------------------- | ----------- | | `loadBalancerClass` *string* | (Optional) | | `allocateLoadBalancerNodePorts` *boolean* | (Optional) | | `loadBalancerSourceRanges` *string array* | (Optional) | | `loadBalancerIP` *string* | (Optional) | ### Goldmane[​](#goldmane) | Field | Description | | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------- | | `apiVersion` *string* | `operator.tigera.io/v1` | | `kind` *string* | `Goldmane` | | `metadata` *[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#objectmeta-v1-meta)* | Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[GoldmaneSpec](#goldmanespec)* | | | `status` *[GoldmaneStatus](#goldmanestatus)* | | ### GoldmaneDeployment[​](#goldmanedeployment) GoldmaneDeployment is the configuration for the goldmane Deployment. *Appears in:* - [GoldmaneSpec](#goldmanespec) | Field | Description | | ---------------------------------------------------------- | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[GoldmaneDeploymentSpec](#goldmanedeploymentspec)* | (Optional) Spec is the specification of the goldmane Deployment. | ### GoldmaneDeploymentContainer[​](#goldmanedeploymentcontainer) *Appears in:* - [GoldmaneDeploymentPodSpec](#goldmanedeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ----------- | | `name` *string* | | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) | ### GoldmaneDeploymentPodSpec[​](#goldmanedeploymentpodspec) GoldmaneDeploymentPodSpec is the goldmane Deployment's PodSpec. *Appears in:* - [GoldmaneDeploymentPodTemplateSpec](#goldmanedeploymentpodtemplatespec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `affinity` *[Affinity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#affinity-v1-core)* | (Optional) Affinity is a group of affinity scheduling rules for the goldmane pods. | | `containers` *[GoldmaneDeploymentContainer](#goldmanedeploymentcontainer) array* | (Optional) Containers is a list of goldmane containers. If specified, this overrides the specified EGW Deployment containers. If omitted, the goldmane Deployment will use its default values for its containers. | | `nodeSelector` *object (keys:string, values:string)* | (Optional) NodeSelector gives more control over the nodes where the goldmane pods will run on. | | `terminationGracePeriodSeconds` *integer* | (Optional) TerminationGracePeriodSeconds defines the termination grace period of the goldmane pods in seconds. | | `topologySpreadConstraints` *[TopologySpreadConstraint](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#topologyspreadconstraint-v1-core) array* | (Optional) TopologySpreadConstraints describes how a group of pods ought to spread across topology domains. Scheduler will schedule pods in a way which abides by the constraints. All topologySpreadConstraints are ANDed. | | `tolerations` *[Toleration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#toleration-v1-core) array* | (Optional) Tolerations is the goldmane pod's tolerations. If specified, this overrides any tolerations that may be set on the goldmane Deployment. If omitted, the goldmane Deployment will use its default value for tolerations. | | `priorityClassName` *string* | (Optional) PriorityClassName allows to specify a PriorityClass resource to be used. | ### GoldmaneDeploymentPodTemplateSpec[​](#goldmanedeploymentpodtemplatespec) GoldmaneDeploymentPodTemplateSpec is the goldmane Deployment's PodTemplateSpec *Appears in:* - [GoldmaneDeploymentSpec](#goldmanedeploymentspec) | Field | Description | | ---------------------------------------------------------------- | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[GoldmaneDeploymentPodSpec](#goldmanedeploymentpodspec)* | (Optional) Spec is the goldmane Deployment's PodSpec. | ### GoldmaneDeploymentSpec[​](#goldmanedeploymentspec) GoldmaneDeploymentSpec defines configuration for the goldmane Deployment. *Appears in:* - [GoldmaneDeployment](#goldmanedeployment) | Field | Description | | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `minReadySeconds` *integer* | (Optional) MinReadySeconds is the minimum number of seconds for which a newly created Deployment pod should be ready without any of its container crashing, for it to be considered available. If specified, this overrides any minReadySeconds value that may be set on the goldmane Deployment. If omitted, the goldmane Deployment will use its default value for minReadySeconds. | | `template` *[GoldmaneDeploymentPodTemplateSpec](#goldmanedeploymentpodtemplatespec)* | (Optional) Template describes the goldmane Deployment pod that will be created. | | `strategy` *[GoldmaneDeploymentStrategy](#goldmanedeploymentstrategy)* | (Optional) The deployment strategy to use to replace existing pods with new ones. | ### GoldmaneDeploymentStrategy[​](#goldmanedeploymentstrategy) *Appears in:* - [GoldmaneDeploymentSpec](#goldmanedeploymentspec) | Field | Description | | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | `rollingUpdate` *[RollingUpdateDeployment](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#rollingupdatedeployment-v1-apps)* | (Optional) Rolling update config params. Present only if DeploymentStrategyType = RollingUpdate. to be. | ### GoldmaneSpec[​](#goldmanespec) *Appears in:* - [Goldmane](#goldmane) | Field | Description | | ---------------------------------------------------------------- | ----------- | | `goldmaneDeployment` *[GoldmaneDeployment](#goldmanedeployment)* | | ### GoldmaneStatus[​](#goldmanestatus) GoldmaneStatus defines the observed state of Goldmane *Appears in:* - [Goldmane](#goldmane) | Field | Description | | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `conditions` *[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#condition-v1-meta) array* | (Optional) Conditions represents the latest observed set of conditions for the component. A component may be one or more of Ready, Progressing, Degraded or other customer types. | ### GroupSearch[​](#groupsearch) Group search configuration to find the groups that a user is in. *Appears in:* - [AuthenticationLDAP](#authenticationldap) | Field | Description | | ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `baseDN` *string* | BaseDN to start the search from. For example "cn=groups,dc=example,dc=com" | | `filter` *string* | (Optional) Optional filter to apply when searching the directory. For example "(objectClass=posixGroup)" | | `nameAttribute` *string* | The attribute of the group that represents its name. This attribute can be used to apply RBAC to a user group. | | `userMatchers` *[UserMatch](#usermatch) array* | Following list contains field pairs that are used to match a user to a group. It adds an additional requirement to the filter that an attribute in the group must match the user's attribute value. | ### GuardianDeployment[​](#guardiandeployment) GuardianDeployment is the configuration for the guardian Deployment. *Appears in:* - [ManagementClusterConnectionSpec](#managementclusterconnectionspec) | Field | Description | | ---------------------------------------------------------- | ---------------------------------------------------------------- | | `spec` *[GuardianDeploymentSpec](#guardiandeploymentspec)* | (Optional) Spec is the specification of the guardian Deployment. | ### GuardianDeploymentContainer[​](#guardiandeploymentcontainer) GuardianDeploymentContainer is a guardian Deployment container. *Appears in:* - [GuardianDeploymentPodSpec](#guardiandeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the guardian Deployment container by name. Supported values are: tigera-guardian | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named guardian Deployment container's resources. If omitted, the guardian Deployment will use its default value for this container's resources. | ### GuardianDeploymentInitContainer[​](#guardiandeploymentinitcontainer) GuardianDeploymentInitContainer is a guardian Deployment init container. *Appears in:* - [GuardianDeploymentPodSpec](#guardiandeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the guardian Deployment init container by name. | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named guardian Deployment init container's resources. If omitted, the guardian Deployment will use its default value for this init container's resources. | ### GuardianDeploymentPodSpec[​](#guardiandeploymentpodspec) GuardianDeploymentPodSpec is the guardian Deployment's PodSpec. *Appears in:* - [GuardianDeploymentPodTemplateSpec](#guardiandeploymentpodtemplatespec) | Field | Description | | -------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[GuardianDeploymentInitContainer](#guardiandeploymentinitcontainer) array* | (Optional) InitContainers is a list of guardian init containers. If specified, this overrides the specified guardian Deployment init containers. If omitted, the guardian Deployment will use its default values for its init containers. | | `containers` *[GuardianDeploymentContainer](#guardiandeploymentcontainer) array* | (Optional) Containers is a list of guardian containers. If specified, this overrides the specified guardian Deployment containers. If omitted, the guardian Deployment will use its default values for its containers. | ### GuardianDeploymentPodTemplateSpec[​](#guardiandeploymentpodtemplatespec) GuardianDeploymentPodTemplateSpec is the guardian Deployment's PodTemplateSpec *Appears in:* - [GuardianDeploymentSpec](#guardiandeploymentspec) | Field | Description | | ---------------------------------------------------------------- | ----------------------------------------------------- | | `spec` *[GuardianDeploymentPodSpec](#guardiandeploymentpodspec)* | (Optional) Spec is the guardian Deployment's PodSpec. | ### GuardianDeploymentSpec[​](#guardiandeploymentspec) GuardianDeploymentSpec defines configuration for the guardian Deployment. *Appears in:* - [GuardianDeployment](#guardiandeployment) | Field | Description | | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------- | | `template` *[GuardianDeploymentPodTemplateSpec](#guardiandeploymentpodtemplatespec)* | (Optional) Template describes the guardian Deployment pod that will be created. | ### HTTPProbe[​](#httpprobe) HTTPProbe defines the HTTP probe configuration for Egress Gateway. *Appears in:* - [EgressGatewayFailureDetection](#egressgatewayfailuredetection) | Field | Description | | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `urls` *string array* | URLs define the list of HTTP probe URLs. Egress Gateway will probe each URL periodically.If all probes fail, Egress Gateway will report non-ready. | | `intervalSeconds` *integer* | (Optional) IntervalSeconds defines the interval of HTTP probes. Used when URLs is non-empty. Default: 10 | | `timeoutSeconds` *integer* | (Optional) TimeoutSeconds defines the timeout value of HTTP probes. Used when URLs is non-empty. Default: 30 | ### HostPortsType[​](#hostportstype) *Underlying type:* *string* HostPortsType specifies host port support. One of: Enabled, Disabled *Appears in:* - [CalicoNetworkSpec](#caliconetworkspec) | Value | Description | | ---------- | ----------- | | `Enabled` | | | `Disabled` | | ### HostScope[​](#hostscope) *Underlying type:* *string* HostScope determines the set of hosts that forward logs to a given store. *Validation:* - Enum: \[All NonClusterOnly] *Appears in:* - [S3StoreSpec](#s3storespec) - [SplunkStoreSpec](#splunkstorespec) - [SyslogStoreSpec](#syslogstorespec) | Value | Description | | ---------------- | ----------- | | `All` | | | `NonClusterOnly` | | ### ICMPProbe[​](#icmpprobe) ICMPProbe defines the ICMP probe configuration for Egress Gateway. *Appears in:* - [EgressGatewayFailureDetection](#egressgatewayfailuredetection) | Field | Description | | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | `ips` *string array* | IPs define the list of ICMP probe IPs. Egress Gateway will probe each IP periodically. If all probes fail, Egress Gateway will report non-ready. | | `intervalSeconds` *integer* | (Optional) IntervalSeconds defines the interval of ICMP probes. Used when IPs is non-empty. Default: 5 | | `timeoutSeconds` *integer* | (Optional) TimeoutSeconds defines the timeout value of ICMP probes. Used when IPs is non-empty. Default: 15 | ### IPAMPluginType[​](#ipamplugintype) *Underlying type:* *string* *Appears in:* - [IPAMSpec](#ipamspec) | Value | Description | | ----------- | ----------- | | `Calico` | | | `HostLocal` | | | `AmazonVPC` | | | `AzureVNET` | | ### IPAMSpec[​](#ipamspec) IPAMSpec contains configuration for pod IP address management. *Appears in:* - [CNISpec](#cnispec) | Field | Description | | ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` *[IPAMPluginType](#ipamplugintype)* | Specifies the IPAM plugin that will be used in the Calico or Calico Enterprise installation. \* For CNI Plugin Calico, this field defaults to Calico. \* For CNI Plugin GKE, this field defaults to HostLocal. \* For CNI Plugin AzureVNET, this field defaults to AzureVNET. \* For CNI Plugin AmazonVPC, this field defaults to AmazonVPC. The IPAM plugin is installed and configured only if the CNI plugin is set to Calico, for all other values of the CNI plugin the plugin binaries and CNI config is a dependency that is expected to be installed separately. Default: Calico | ### IPPool[​](#ippool) *Appears in:* - [CalicoNetworkSpec](#caliconetworkspec) | Field | Description | | ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is the name of the IP pool. If omitted, this will be generated. | | `cidr` *string* | CIDR contains the address range for the IP Pool in classless inter-domain routing format. | | `encapsulation` *[EncapsulationType](#encapsulationtype)* | (Optional) Encapsulation specifies the encapsulation type that will be used with the IP Pool. Default: IPIP | | `natOutgoing` *[NATOutgoingType](#natoutgoingtype)* | (Optional) NATOutgoing specifies if NAT will be enabled or disabled for outgoing traffic. Default: Enabled | | `nodeSelector` *string* | (Optional) NodeSelector specifies the node selector that will be set for the IP Pool. Default: 'all()' | | `blockSize` *integer* | (Optional) BlockSize specifies the CIDR prefex length to use when allocating per-node IP blocks from the main IP pool CIDR. Default: 26 (IPv4), 122 (IPv6) | | `disableBGPExport` *boolean* | (Optional) DisableBGPExport specifies whether routes from this IP pool's CIDR are exported over BGP. Default: false | | `disableNewAllocations` *boolean* | DisableNewAllocations specifies whether or not new IP allocations are allowed from this pool. This is useful when you want to prevent new pods from receiving IP addresses from this pool, without impacting any existing pods that have already been assigned addresses from this pool. | | `allowedUses` *[IPPoolAllowedUse](#ippoolalloweduse) array* | AllowedUse controls what the IP pool will be used for. If not specified or empty, defaults to \["Tunnel", "Workload"] for back-compatibility | | `assignmentMode` *[AssignmentMode](#assignmentmode)* | AssignmentMode determines if IP addresses from this pool should be assigned automatically or on request only | ### IPPoolAllowedUse[​](#ippoolalloweduse) *Underlying type:* *string* *Appears in:* - [IPPool](#ippool) | Value | Description | | -------------- | ----------- | | `Workload` | | | `Tunnel` | | | `LoadBalancer` | | ### Image[​](#image) *Appears in:* - [ImageSetSpec](#imagesetspec) | Field | Description | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `image` *string* | Image is an image that the operator deploys and instead of using the built in tag the operator will use the Digest for the image identifier. The value should be the *original* image name without registry or tag or digest. For the image `docker.io/calico/node:v3.17.1` it should be represented as `calico/node` The "Installation" spec allows defining custom image registries, paths or prefixes. Even for custom images such as example.com/custompath/customprefix-calico-node:v3.17.1, this value should still be `calico/node`. | | `digest` *string* | Digest is the image identifier that will be used for the Image. The field should not include a leading `@` and must be prefixed with `sha256:`. | ### ImageSet[​](#imageset) ImageSet is used to specify image digests for the images that the operator deploys. The name of the ImageSet is expected to be in the format `-`. The `variant` used is `enterprise` if the InstallationSpec Variant is `TigeraSecureEnterprise` otherwise it is `calico`. The `release` must match the version of the variant that the operator is built to deploy, this version can be obtained by passing the `--version` flag to the operator binary. | Field | Description | | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------- | | `apiVersion` *string* | `operator.tigera.io/v1` | | `kind` *string* | `ImageSet` | | `metadata` *[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#objectmeta-v1-meta)* | Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[ImageSetSpec](#imagesetspec)* | | ### ImageSetSpec[​](#imagesetspec) ImageSetSpec defines the desired state of ImageSet. *Appears in:* - [ImageSet](#imageset) | Field | Description | | -------------------------------- | -------------------------------------------------------------------------------------------------------- | | `images` *[Image](#image) array* | Images is the list of images to use digests. All images that the operator will deploy must be specified. | ### Impersonation[​](#impersonation) Impersonation defines the rules for allowing impersonation. *Appears in:* - [ManagementClusterConnectionSpec](#managementclusterconnectionspec) | Field | Description | | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `users` *string array* | (Optional) Users is a list of users that can be impersonated. An empty list infers all users can be impersonated, a null value means none. | | `groups` *string array* | (Optional) Groups is a list of group names that can be impersonated. An empty list infers all groups can be impersonated, a null values means none. | | `serviceAccounts` *string array* | (Optional) ServiceAccounts is a list of service account names that can be impersonated. An empty list infers all service accounts can be impersonated, a null values means none. | ### Indices[​](#indices) Indices defines the configuration for the indices in an Elasticsearch cluster. *Appears in:* - [LogStorageSpec](#logstoragespec) | Field | Description | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `replicas` *integer* | (Optional) Replicas defines how many replicas each index will have. See [https://www.elastic.co/guide/en/elasticsearch/reference/current/scalability.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/scalability.html) | ### Installation[​](#installation) Installation configures an installation of Calico or Calico Enterprise. At most one instance of this resource is supported. It must be named "default". The Installation API installs core networking and network policy components, and provides general install-time configuration. | Field | Description | | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------ | | `apiVersion` *string* | `operator.tigera.io/v1` | | `kind` *string* | `Installation` | | `metadata` *[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#objectmeta-v1-meta)* | Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[InstallationSpec](#installationspec)* | Specification of the desired state for the Calico or Calico Enterprise installation. | | `status` *[InstallationStatus](#installationstatus)* | Most recently observed state for the Calico or Calico Enterprise installation. | ### InstallationSpec[​](#installationspec) InstallationSpec defines configuration for a Calico or Calico Enterprise installation. *Appears in:* - [Installation](#installation) - [InstallationStatus](#installationstatus) | Field | Description | | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `variant` *[ProductVariant](#productvariant)* | (Optional) Variant is the product to install - one of Calico or TigeraSecureEnterprise Default: Calico | | `registry` *string* | (Optional) Registry is the default Docker registry used for component Docker images. If specified then the given value must end with a slash character (`/`) and all images will be pulled from this registry. If not specified then the default registries will be used. A special case value, UseDefault, is supported to explicitly specify the default registries will be used. Image format: `/:` This option allows configuring the `` portion of the above format. | | `imagePath` *string* | (Optional) ImagePath allows for the path part of an image to be specified. If specified then the specified value will be used as the image path for each image. If not specified or empty, the default for each image will be used. A special case value, UseDefault, is supported to explicitly specify the default image path will be used for each image. Image format: `/:` This option allows configuring the `` portion of the above format. | | `imagePrefix` *string* | (Optional) ImagePrefix allows for the prefix part of an image to be specified. If specified then the given value will be used as a prefix on each image. If not specified or empty, no prefix will be used. A special case value, UseDefault, is supported to explicitly specify the default image prefix will be used for each image. Image format: `/:` This option allows configuring the `` portion of the above format. | | `imagePullSecrets` *[LocalObjectReference](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#localobjectreference-v1-core) array* | (Optional) ImagePullSecrets is an array of references to container registry pull secrets to use. These are applied to all images to be pulled. | | `kubernetesProvider` *[Provider](#provider)* | (Optional) KubernetesProvider specifies a particular provider of the Kubernetes platform and enables provider-specific configuration. If the specified value is empty, the Operator will attempt to automatically determine the current provider. If the specified value is not empty, the Operator will still attempt auto-detection, but will additionally compare the auto-detected value to the specified value to confirm they match. | | `cni` *[CNISpec](#cnispec)* | (Optional) CNI specifies the CNI that will be used by this installation. | | `calicoNetwork` *[CalicoNetworkSpec](#caliconetworkspec)* | (Optional) CalicoNetwork specifies networking configuration options for Calico. | | `typhaAffinity` *[TyphaAffinity](#typhaaffinity)* | (Optional) Deprecated. Please use Installation.Spec.TyphaDeployment instead. TyphaAffinity allows configuration of node affinity characteristics for Typha pods. | | `controlPlaneNodeSelector` *object (keys:string, values:string)* | (Optional) ControlPlaneNodeSelector is used to select control plane nodes on which to run Calico components. This is globally applied to all resources created by the operator excluding daemonsets. | | `controlPlaneTolerations` *[Toleration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#toleration-v1-core) array* | (Optional) ControlPlaneTolerations specify tolerations which are then globally applied to all resources created by the operator. | | `controlPlaneReplicas` *integer* | (Optional) ControlPlaneReplicas defines how many replicas of the control plane core components will be deployed. This field applies to all control plane components that support High Availability. Defaults to 2. | | `nodeMetricsPort` *integer* | (Optional) NodeMetricsPort specifies which port calico/node serves prometheus metrics on. By default, metrics are not enabled. If specified, this overrides any FelixConfiguration resources which may exist. If omitted, then prometheus metrics may still be configured through FelixConfiguration. | | `typhaMetricsPort` *integer* | (Optional) TyphaMetricsPort specifies which port calico/typha serves prometheus metrics on. By default, metrics are not enabled. | | `flexVolumePath` *string* | (Optional) FlexVolumePath optionally specifies a custom path for FlexVolume. If not specified, FlexVolume will be enabled by default. If set to 'None', FlexVolume will be disabled. The default is based on the kubernetesProvider. | | `kubeletVolumePluginPath` *string* | (Optional) KubeletVolumePluginPath optionally specifies enablement of Calico CSI plugin. If not specified, CSI will be enabled by default. If set to 'None', CSI will be disabled. Default: /var/lib/kubelet | | `nodeUpdateStrategy` *[DaemonSetUpdateStrategy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#daemonsetupdatestrategy-v1-apps)* | (Optional) NodeUpdateStrategy can be used to customize the desired update strategy, such as the MaxUnavailable field. | | `componentResources` *[ComponentResource](#componentresource) array* | (Optional) Deprecated. Please use CalicoNodeDaemonSet, TyphaDeployment, and KubeControllersDeployment. ComponentResources can be used to customize the resource requirements for each component. Node, Typha, and KubeControllers are supported for installations. | | `certificateManagement` *[CertificateManagement](#certificatemanagement)* | (Optional) CertificateManagement configures pods to submit a CertificateSigningRequest to the certificates.k8s.io/v1 API in order to obtain TLS certificates. This feature requires that you bring your own CSR signing and approval process, otherwise pods will be stuck during initialization. | | `tlsCipherSuites` *[TLSCipherSuites](#tlsciphersuites)* | (Optional) TLSCipherSuites defines the cipher suite list that the TLS protocol should use during secure communication. | | `nonPrivileged` *[NonPrivilegedType](#nonprivilegedtype)* | (Optional) NonPrivileged configures Calico to be run in non-privileged containers as non-root users where possible. | | `calicoNodeDaemonSet` *[CalicoNodeDaemonSet](#caliconodedaemonset)* | (Optional) CalicoNodeDaemonSet configures the calico-node DaemonSet. If used in conjunction with the deprecated ComponentResources, then these overrides take precedence. | | `csiNodeDriverDaemonSet` *[CSINodeDriverDaemonSet](#csinodedriverdaemonset)* | (Optional) CSINodeDriverDaemonSet configures the csi-node-driver DaemonSet. | | `calicoKubeControllersDeployment` *[CalicoKubeControllersDeployment](#calicokubecontrollersdeployment)* | (Optional) CalicoKubeControllersDeployment configures the calico-kube-controllers Deployment. If used in conjunction with the deprecated ComponentResources, then these overrides take precedence. | | `typhaDeployment` *[TyphaDeployment](#typhadeployment)* | (Optional) TyphaDeployment configures the typha Deployment. If used in conjunction with the deprecated ComponentResources or TyphaAffinity, then these overrides take precedence. | | `calicoWindowsUpgradeDaemonSet` *[CalicoWindowsUpgradeDaemonSet](#calicowindowsupgradedaemonset)* | Deprecated. The CalicoWindowsUpgradeDaemonSet is deprecated and will be removed from the API in the future. CalicoWindowsUpgradeDaemonSet configures the calico-windows-upgrade DaemonSet. | | `calicoNodeWindowsDaemonSet` *[CalicoNodeWindowsDaemonSet](#caliconodewindowsdaemonset)* | CalicoNodeWindowsDaemonSet configures the calico-node-windows DaemonSet. | | `fipsMode` *[FIPSMode](#fipsmode)* | (Optional) FIPSMode uses images and features only that are using FIPS 140-2 validated cryptographic modules and standards. Only supported for Variant=Calico. Default: Disabled | | `logging` *[Logging](#logging)* | (Optional) Logging Configuration for Components | | `windowsNodes` *[WindowsNodeSpec](#windowsnodespec)* | (Optional) Windows Configuration | | `serviceCIDRs` *string array* | (Optional) Kubernetes Service CIDRs. Specifying this is required when using Calico for Windows. | | `azure` *[Azure](#azure)* | (Optional) Azure is used to configure azure provider specific options. | | `proxy` *[Proxy](#proxy)* | (Optional) Proxy is used to configure the HTTP(S) proxy settings that will be applied to Tigera containers that connect to destinations outside the cluster. It is expected that NO\_PROXY is configured such that destinations within the cluster (including the API server) are exempt from proxying. | ### InstallationStatus[​](#installationstatus) InstallationStatus defines the observed state of the Calico or Calico Enterprise installation. *Appears in:* - [Installation](#installation) | Field | Description | | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `variant` *[ProductVariant](#productvariant)* | Variant is the most recently observed installed variant - one of Calico or TigeraSecureEnterprise | | `mtu` *integer* | MTU is the most recently observed value for pod network MTU. This may be an explicitly configured value, or based on Calico's native auto-detetion. | | `imageSet` *string* | (Optional) ImageSet is the name of the ImageSet being used, if there is an ImageSet that is being used. If an ImageSet is not being used then this will not be set. | | `computed` *[InstallationSpec](#installationspec)* | (Optional) Computed is the final installation including overlaid resources. | | `calicoVersion` *string* | CalicoVersion shows the current running version of calico. CalicoVersion along with Variant is needed to know the exact version deployed. | | `conditions` *[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#condition-v1-meta) array* | (Optional) Conditions represents the latest observed set of conditions for the component. A component may be one or more of Ready, Progressing, Degraded or other customer types. | ### IntrusionDetection[​](#intrusiondetection) IntrusionDetection installs the components required for Tigera intrusion detection. At most one instance of this resource is supported. It must be named "tigera-secure". | Field | Description | | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------ | | `apiVersion` *string* | `operator.tigera.io/v1` | | `kind` *string* | `IntrusionDetection` | | `metadata` *[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#objectmeta-v1-meta)* | Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[IntrusionDetectionSpec](#intrusiondetectionspec)* | Specification of the desired state for Tigera intrusion detection. | | `status` *[IntrusionDetectionStatus](#intrusiondetectionstatus)* | Most recently observed state for Tigera intrusion detection. | ### IntrusionDetectionComponentName[​](#intrusiondetectioncomponentname) *Underlying type:* *string* *Appears in:* - [IntrusionDetectionComponentResource](#intrusiondetectioncomponentresource) | Value | Description | | ---------------------- | ----------- | | `DeepPacketInspection` | | ### IntrusionDetectionComponentResource[​](#intrusiondetectioncomponentresource) The ComponentResource struct associates a ResourceRequirements with a component by name *Appears in:* - [IntrusionDetectionSpec](#intrusiondetectionspec) | Field | Description | | -------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | `componentName` *[IntrusionDetectionComponentName](#intrusiondetectioncomponentname)* | ComponentName is an enum which identifies the component | | `resourceRequirements` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | ResourceRequirements allows customization of limits and requests for compute resources such as cpu and memory. | ### IntrusionDetectionControllerDeployment[​](#intrusiondetectioncontrollerdeployment) IntrusionDetectionControllerDeployment is the configuration for the IntrusionDetectionController Deployment. *Appears in:* - [IntrusionDetectionSpec](#intrusiondetectionspec) | Field | Description | | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | | `spec` *[IntrusionDetectionControllerDeploymentSpec](#intrusiondetectioncontrollerdeploymentspec)* | (Optional) Spec is the specification of the IntrusionDetectionController Deployment. | ### IntrusionDetectionControllerDeploymentContainer[​](#intrusiondetectioncontrollerdeploymentcontainer) IntrusionDetectionControllerDeploymentContainer is a IntrusionDetectionController Deployment container. *Appears in:* - [IntrusionDetectionControllerDeploymentPodSpec](#intrusiondetectioncontrollerdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the IntrusionDetectionController Deployment container by name. Supported values are: controller, webhooks-processor | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named IntrusionDetectionController Deployment container's resources. If omitted, the IntrusionDetection Deployment will use its default value for this container's resources. | ### IntrusionDetectionControllerDeploymentInitContainer[​](#intrusiondetectioncontrollerdeploymentinitcontainer) IntrusionDetectionControllerDeploymentInitContainer is a IntrusionDetectionController Deployment init container. *Appears in:* - [IntrusionDetectionControllerDeploymentPodSpec](#intrusiondetectioncontrollerdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the IntrusionDetectionController Deployment init container by name. Supported values are: intrusion-detection-tls-key-cert-provisioner | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named IntrusionDetectionController Deployment init container's resources. If omitted, the IntrusionDetectionController Deployment will use its default value for this init container's resources. | ### IntrusionDetectionControllerDeploymentPodSpec[​](#intrusiondetectioncontrollerdeploymentpodspec) IntrusionDetectionControllerDeploymentPodSpec is the IntrusionDetectionController Deployment's PodSpec. *Appears in:* - [IntrusionDetectionControllerDeploymentPodTemplateSpec](#intrusiondetectioncontrollerdeploymentpodtemplatespec) | Field | Description | | ------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[IntrusionDetectionControllerDeploymentInitContainer](#intrusiondetectioncontrollerdeploymentinitcontainer) array* | (Optional) InitContainers is a list of IntrusionDetectionController init containers. If specified, this overrides the specified IntrusionDetectionController Deployment init containers. If omitted, the IntrusionDetectionController Deployment will use its default values for its init containers. | | `containers` *[IntrusionDetectionControllerDeploymentContainer](#intrusiondetectioncontrollerdeploymentcontainer) array* | (Optional) Containers is a list of IntrusionDetectionController containers. If specified, this overrides the specified IntrusionDetectionController Deployment containers. If omitted, the IntrusionDetectionController Deployment will use its default values for its containers. | ### IntrusionDetectionControllerDeploymentPodTemplateSpec[​](#intrusiondetectioncontrollerdeploymentpodtemplatespec) IntrusionDetectionControllerDeploymentPodTemplateSpec is the IntrusionDetectionController Deployment's PodTemplateSpec *Appears in:* - [IntrusionDetectionControllerDeploymentSpec](#intrusiondetectioncontrollerdeploymentspec) | Field | Description | | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | | `spec` *[IntrusionDetectionControllerDeploymentPodSpec](#intrusiondetectioncontrollerdeploymentpodspec)* | (Optional) Spec is the IntrusionDetectionController Deployment's PodSpec. | ### IntrusionDetectionControllerDeploymentSpec[​](#intrusiondetectioncontrollerdeploymentspec) IntrusionDetectionControllerDeploymentSpec defines configuration for the IntrusionDetectionController Deployment. *Appears in:* - [IntrusionDetectionControllerDeployment](#intrusiondetectioncontrollerdeployment) | Field | Description | | ---------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | | `template` *[IntrusionDetectionControllerDeploymentPodTemplateSpec](#intrusiondetectioncontrollerdeploymentpodtemplatespec)* | (Optional) Template describes the IntrusionDetectionController Deployment pod that will be created. | ### IntrusionDetectionSpec[​](#intrusiondetectionspec) IntrusionDetectionSpec defines the desired state of Tigera intrusion detection capabilities. *Appears in:* - [IntrusionDetection](#intrusiondetection) | Field | Description | | ---------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `componentResources` *[IntrusionDetectionComponentResource](#intrusiondetectioncomponentresource) array* | (Optional) ComponentResources can be used to customize the resource requirements for each component. Only DeepPacketInspection is supported for this spec. | | `anomalyDetection` *[AnomalyDetectionSpec](#anomalydetectionspec)* | (Optional) AnomalyDetection is now deprecated, and configuring it has no effect. | | `intrusionDetectionControllerDeployment` *[IntrusionDetectionControllerDeployment](#intrusiondetectioncontrollerdeployment)* | (Optional) IntrusionDetectionControllerDeployment configures the IntrusionDetection Controller Deployment. | | `deepPacketInspectionDaemonset` *[DeepPacketInspectionDaemonset](#deeppacketinspectiondaemonset)* | (Optional) DeepPacketInspectionDaemonset configures the DPI Daemonset | ### IntrusionDetectionStatus[​](#intrusiondetectionstatus) IntrusionDetectionStatus defines the observed state of Tigera intrusion detection capabilities. *Appears in:* - [IntrusionDetection](#intrusiondetection) | Field | Description | | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `state` *string* | State provides user-readable status. | | `conditions` *[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#condition-v1-meta) array* | (Optional) Conditions represents the latest observed set of conditions for the component. A component may be one or more of Ready, Progressing, Degraded or other customer types. | ### Kibana[​](#kibana) Kibana is the configuration for the Kibana. *Appears in:* - [LogStorageSpec](#logstoragespec) | Field | Description | | ---------------------------------- | --------------------------------------------------- | | `spec` *[KibanaSpec](#kibanaspec)* | (Optional) Spec is the specification of the Kibana. | ### KibanaContainer[​](#kibanacontainer) KibanaContainer is a Kibana container. *Appears in:* - [KibanaPodSpec](#kibanapodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the Kibana Deployment container by name. Supported values are: kibana | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named Kibana container's resources. If omitted, the Kibana will use its default value for this container's resources. | ### KibanaInitContainer[​](#kibanainitcontainer) KibanaInitContainer is a Kibana init container. *Appears in:* - [KibanaPodSpec](#kibanapodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `name` *string* | Name is an enum which identifies the Kibana init container by name. Supported values are: key-cert-provisioner | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named Kibana Deployment init container's resources. If omitted, the Kibana Deployment will use its default value for this init container's resources. If used in conjunction with the deprecated ComponentResources, then this value takes precedence. | ### KibanaPodSpec[​](#kibanapodspec) KibanaPodSpec is the Kibana Deployment's PodSpec. *Appears in:* - [KibanaPodTemplateSpec](#kibanapodtemplatespec) | Field | Description | | -------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[KibanaInitContainer](#kibanainitcontainer) array* | (Optional) InitContainers is a list of Kibana init containers. If specified, this overrides the specified Kibana Deployment init containers. If omitted, the Kibana Deployment will use its default values for its init containers. | | `containers` *[KibanaContainer](#kibanacontainer) array* | (Optional) Containers is a list of Kibana containers. If specified, this overrides the specified Kibana Deployment containers. If omitted, the Kibana Deployment will use its default values for its containers. | ### KibanaPodTemplateSpec[​](#kibanapodtemplatespec) KibanaPodTemplateSpec is the Kibana's PodTemplateSpec *Appears in:* - [KibanaSpec](#kibanaspec) | Field | Description | | ---------------------------------------- | ---------------------------------------- | | `spec` *[KibanaPodSpec](#kibanapodspec)* | (Optional) Spec is the Kibana's PodSpec. | ### KibanaSpec[​](#kibanaspec) *Appears in:* - [Kibana](#kibana) | Field | Description | | ------------------------------------------------------------ | ------------------------------------------------------------------ | | `template` *[KibanaPodTemplateSpec](#kibanapodtemplatespec)* | (Optional) Template describes the Kibana pod that will be created. | ### KubeProxyManagementType[​](#kubeproxymanagementtype) *Underlying type:* *string* KubeProxyManagementType specifies whether kube-proxy management is enabled. *Appears in:* - [CalicoNetworkSpec](#caliconetworkspec) | Value | Description | | ---------- | ----------- | | `Enabled` | | | `Disabled` | | ### KubernetesAutodetectionMethod[​](#kubernetesautodetectionmethod) *Underlying type:* *string* KubernetesAutodetectionMethod is a method of detecting an IP address based on the Kubernetes API. One of: NodeInternalIP *Appears in:* - [NodeAddressAutodetection](#nodeaddressautodetection) | Value | Description | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | `NodeInternalIP` | NodeInternalIP detects a node IP using the first status.Addresses entry of the relevant IP family with type NodeInternalIP on the Kubernetes nodes API. | ### L7LogCollectorDaemonSet[​](#l7logcollectordaemonset) L7LogCollectorDaemonSet is the configuration for the L7LogCollector DaemonSet. *Appears in:* - [ApplicationLayerSpec](#applicationlayerspec) | Field | Description | | -------------------------------------------------------------------- | --------------------------------------------------------------------- | | `spec` *[L7LogCollectorDaemonSetSpec](#l7logcollectordaemonsetspec)* | (Optional) Spec is the specification of the L7LogCollector DaemonSet. | ### L7LogCollectorDaemonSetContainer[​](#l7logcollectordaemonsetcontainer) L7LogCollectorDaemonSetContainer is a L7LogCollector DaemonSet container. *Appears in:* - [L7LogCollectorDaemonSetPodSpec](#l7logcollectordaemonsetpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the L7LogCollector DaemonSet container by name. Supported values are: l7-collector, envoy-proxy, dikastes | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named L7LogCollector DaemonSet container's resources. If omitted, the L7LogCollector DaemonSet will use its default value for this container's resources. | ### L7LogCollectorDaemonSetInitContainer[​](#l7logcollectordaemonsetinitcontainer) L7LogCollectorDaemonSetInitContainer is a L7LogCollector DaemonSet init container. *Appears in:* - [L7LogCollectorDaemonSetPodSpec](#l7logcollectordaemonsetpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the L7LogCollector DaemonSet init container by name. | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named L7LogCollector DaemonSet init container's resources. If omitted, the L7LogCollector DaemonSet will use its default value for this init container's resources. | ### L7LogCollectorDaemonSetPodSpec[​](#l7logcollectordaemonsetpodspec) L7LogCollectorDaemonSetPodSpec is the L7LogCollector DaemonSet's PodSpec. *Appears in:* - [L7LogCollectorDaemonSetPodTemplateSpec](#l7logcollectordaemonsetpodtemplatespec) | Field | Description | | ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[L7LogCollectorDaemonSetInitContainer](#l7logcollectordaemonsetinitcontainer) array* | (Optional) InitContainers is a list of L7LogCollector DaemonSet init containers. If specified, this overrides the specified L7LogCollector DaemonSet init containers. If omitted, the L7LogCollector DaemonSet will use its default values for its init containers. | | `containers` *[L7LogCollectorDaemonSetContainer](#l7logcollectordaemonsetcontainer) array* | (Optional) Containers is a list of L7LogCollector DaemonSet containers. If specified, this overrides the specified L7LogCollector DaemonSet containers. If omitted, the L7LogCollector DaemonSet will use its default values for its containers. | ### L7LogCollectorDaemonSetPodTemplateSpec[​](#l7logcollectordaemonsetpodtemplatespec) L7LogCollectorDaemonSetPodTemplateSpec is the L7LogCollector DaemonSet's PodTemplateSpec *Appears in:* - [L7LogCollectorDaemonSetSpec](#l7logcollectordaemonsetspec) | Field | Description | | -------------------------------------------------------------------------- | ---------------------------------------------------------- | | `spec` *[L7LogCollectorDaemonSetPodSpec](#l7logcollectordaemonsetpodspec)* | (Optional) Spec is the L7LogCollector DaemonSet's PodSpec. | ### L7LogCollectorDaemonSetSpec[​](#l7logcollectordaemonsetspec) L7LogCollectorDaemonSetSpec defines configuration for the L7LogCollector DaemonSet. *Appears in:* - [L7LogCollectorDaemonSet](#l7logcollectordaemonset) | Field | Description | | ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | | `template` *[L7LogCollectorDaemonSetPodTemplateSpec](#l7logcollectordaemonsetpodtemplatespec)* | (Optional) Template describes the L7LogCollector DaemonSet pod that will be created. | ### LinseedDeployment[​](#linseeddeployment) LinseedDeployment is the configuration for the linseed Deployment. *Appears in:* - [LogStorageSpec](#logstoragespec) | Field | Description | | -------------------------------------------------------- | --------------------------------------------------------------- | | `spec` *[LinseedDeploymentSpec](#linseeddeploymentspec)* | (Optional) Spec is the specification of the linseed Deployment. | ### LinseedDeploymentContainer[​](#linseeddeploymentcontainer) LinseedDeploymentContainer is a linseed Deployment container. *Appears in:* - [LinseedDeploymentPodSpec](#linseeddeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the linseed Deployment container by name. Supported values are: tigera-linseed | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named linseed Deployment container's resources. If omitted, the linseed Deployment will use its default value for this container's resources. | ### LinseedDeploymentInitContainer[​](#linseeddeploymentinitcontainer) LinseedDeploymentInitContainer is a linseed Deployment init container. *Appears in:* - [LinseedDeploymentPodSpec](#linseeddeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the linseed Deployment init container by name. Supported values are: tigera-secure-linseed-token-tls-key-cert-provisioner,tigera-secure-linseed-cert-key-cert-provisioner | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named linseed Deployment init container's resources. If omitted, the linseed Deployment will use its default value for this init container's resources. | ### LinseedDeploymentPodSpec[​](#linseeddeploymentpodspec) LinseedDeploymentPodSpec is the linseed Deployment's PodSpec. *Appears in:* - [LinseedDeploymentPodTemplateSpec](#linseeddeploymentpodtemplatespec) | Field | Description | | ------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[LinseedDeploymentInitContainer](#linseeddeploymentinitcontainer) array* | (Optional) InitContainers is a list of linseed init containers. If specified, this overrides the specified linseed Deployment init containers. If omitted, the linseed Deployment will use its default values for its init containers. | | `containers` *[LinseedDeploymentContainer](#linseeddeploymentcontainer) array* | (Optional) Containers is a list of linseed containers. If specified, this overrides the specified linseed Deployment containers. If omitted, the linseed Deployment will use its default values for its containers. | ### LinseedDeploymentPodTemplateSpec[​](#linseeddeploymentpodtemplatespec) LinseedDeploymentPodTemplateSpec is the linseed Deployment's PodTemplateSpec *Appears in:* - [LinseedDeploymentSpec](#linseeddeploymentspec) | Field | Description | | -------------------------------------------------------------- | ---------------------------------------------------- | | `spec` *[LinseedDeploymentPodSpec](#linseeddeploymentpodspec)* | (Optional) Spec is the linseed Deployment's PodSpec. | ### LinseedDeploymentSpec[​](#linseeddeploymentspec) LinseedDeploymentSpec defines configuration for the linseed Deployment. *Appears in:* - [LinseedDeployment](#linseeddeployment) | Field | Description | | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | | `template` *[LinseedDeploymentPodTemplateSpec](#linseeddeploymentpodtemplatespec)* | (Optional) Template describes the linseed Deployment pod that will be created. | ### LinuxDataplaneOption[​](#linuxdataplaneoption) *Underlying type:* *string* LinuxDataplaneOption controls which dataplane is to be used on Linux nodes. One of: Iptables, BPF, VPP, Nftables *Validation:* - Enum: \[Iptables BPF VPP Nftables] *Appears in:* - [CalicoNetworkSpec](#caliconetworkspec) | Value | Description | | ---------- | ----------- | | `Iptables` | | | `BPF` | | | `VPP` | | | `Nftables` | | ### LogCollectionSpec[​](#logcollectionspec) *Appears in:* - [ApplicationLayerSpec](#applicationlayerspec) | Field | Description | | ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `collectLogs` *[LogCollectionStatusType](#logcollectionstatustype)* | (Optional) This setting enables or disable log collection. Allowed values are Enabled or Disabled. | | `logIntervalSeconds` *integer* | (Optional) Interval in seconds for sending L7 log information for processing. Default: 5 sec | | `logRequestsPerInterval` *integer* | (Optional) Maximum number of unique L7 logs that are sent LogIntervalSeconds. Adjust this to limit the number of L7 logs sent per LogIntervalSeconds to felix for further processing, use negative number to ignore limits. Default: -1 | ### LogCollectionStatusType[​](#logcollectionstatustype) *Underlying type:* *string* *Validation:* - Enum: \[Enabled Disabled] *Appears in:* - [LogCollectionSpec](#logcollectionspec) | Value | Description | | ---------- | ----------- | | `Disabled` | | | `Enabled` | | ### LogCollector[​](#logcollector) LogCollector installs the components required for Tigera flow and DNS log collection. At most one instance of this resource is supported. It must be named "tigera-secure". When created, this installs fluentd on all nodes configured to collect Tigera log data and export it to Tigera's Elasticsearch cluster as well as any additionally configured destinations. | Field | Description | | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------- | | `apiVersion` *string* | `operator.tigera.io/v1` | | `kind` *string* | `LogCollector` | | `metadata` *[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#objectmeta-v1-meta)* | Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[LogCollectorSpec](#logcollectorspec)* | Specification of the desired state for Tigera log collection. | | `status` *[LogCollectorStatus](#logcollectorstatus)* | Most recently observed state for Tigera log collection. | ### LogCollectorSpec[​](#logcollectorspec) LogCollectorSpec defines the desired state of Tigera flow, audit, and DNS log collection. *Appears in:* - [LogCollector](#logcollector) | Field | Description | | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `additionalStores` *[AdditionalLogStoreSpec](#additionallogstorespec)* | (Optional) Configuration for exporting flow, audit, and DNS logs to external storage. | | `additionalSources` *[AdditionalLogSourceSpec](#additionallogsourcespec)* | (Optional) Configuration for importing audit logs from managed kubernetes cluster log sources. | | `collectProcessPath` *[CollectProcessPathOption](#collectprocesspathoption)* | (Optional) Configuration for enabling/disabling process path collection in flowlogs. If Enabled, this feature sets hostPID to true in order to read process cmdline. Default: Enabled | | `multiTenantManagementClusterNamespace` *string* | (Optional) If running as a multi-tenant management cluster, the namespace in which the management cluster's tenant services are running. | | `fluentdDaemonSet` *[FluentdDaemonSet](#fluentddaemonset)* | FluentdDaemonSet configures the Fluentd DaemonSet. | | `eksLogForwarderDeployment` *[EKSLogForwarderDeployment](#ekslogforwarderdeployment)* | (Optional) EKSLogForwarderDeployment configures the EKSLogForwarderDeployment Deployment. | ### LogCollectorStatus[​](#logcollectorstatus) LogCollectorStatus defines the observed state of Tigera flow and DNS log collection *Appears in:* - [LogCollector](#logcollector) | Field | Description | | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `state` *string* | State provides user-readable status. | | `conditions` *[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#condition-v1-meta) array* | (Optional) Conditions represents the latest observed set of conditions for the component. A component may be one or more of Ready, Progressing, Degraded or other customer types. | ### LogLevel[​](#loglevel) *Underlying type:* *string* *Validation:* - Enum: \[Error Warning Info Debug] *Appears in:* - [CNILogging](#cnilogging) | Value | Description | | ------- | ----------- | | `Error` | | | `Warn` | | | `Info` | | | `Debug` | | ### LogSeverity[​](#logseverity) *Underlying type:* *string* *Validation:* - Enum: \[Fatal Error Warn Info Debug Trace] *Appears in:* - [APIServerLogging](#apiserverlogging) - [EgressGatewaySpec](#egressgatewayspec) - [QueryServerLogging](#queryserverlogging) | Value | Description | | ------- | ----------- | | `Fatal` | | | `Error` | | | `Warn` | | | `Info` | | | `Debug` | | | `Trace` | | ### LogStorage[​](#logstorage) LogStorage installs the components required for Tigera flow and DNS log storage. At most one instance of this resource is supported. It must be named "tigera-secure". When created, this installs an Elasticsearch cluster for use by Calico Enterprise. | Field | Description | | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------- | | `apiVersion` *string* | `operator.tigera.io/v1` | | `kind` *string* | `LogStorage` | | `metadata` *[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#objectmeta-v1-meta)* | Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[LogStorageSpec](#logstoragespec)* | Specification of the desired state for Tigera log storage. | | `status` *[LogStorageStatus](#logstoragestatus)* | Most recently observed state for Tigera log storage. | ### LogStorageComponentName[​](#logstoragecomponentname) *Underlying type:* *string* LogStorageComponentName CRD enum *Appears in:* - [LogStorageComponentResource](#logstoragecomponentresource) | Value | Description | | ------------- | ----------- | | `ECKOperator` | | ### LogStorageComponentResource[​](#logstoragecomponentresource) The ComponentResource struct associates a ResourceRequirements with a component by name *Appears in:* - [LogStorageSpec](#logstoragespec) | Field | Description | | -------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | `componentName` *[LogStorageComponentName](#logstoragecomponentname)* | Deprecated. Please use ECKOperatorStatefulSet. ComponentName is an enum which identifies the component | | `resourceRequirements` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | ResourceRequirements allows customization of limits and requests for compute resources such as cpu and memory. | ### LogStorageSpec[​](#logstoragespec) LogStorageSpec defines the desired state of Tigera flow and DNS log storage. *Appears in:* - [LogStorage](#logstorage) | Field | Description | | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `nodes` *[Nodes](#nodes)* | Nodes defines the configuration for a set of identical Elasticsearch cluster nodes, each of type master, data, and ingest. | | `indices` *[Indices](#indices)* | (Optional) Index defines the configuration for the indices in the Elasticsearch cluster. | | `retention` *[Retention](#retention)* | (Optional) Retention defines how long data is retained in the Elasticsearch cluster before it is cleared. | | `storageClassName` *string* | (Optional) StorageClassName will populate the PersistentVolumeClaim.StorageClassName that is used to provision disks to the Tigera Elasticsearch cluster. The StorageClassName should only be modified when no LogStorage is currently active. We recommend choosing a storage class dedicated to Tigera LogStorage only. Otherwise, data retention cannot be guaranteed during upgrades. See [https://docs.tigera.io/maintenance/upgrading](https://docs.tigera.io/maintenance/upgrading) for up-to-date instructions. Default: tigera-elasticsearch | | `dataNodeSelector` *object (keys:string, values:string)* | (Optional) DataNodeSelector gives you more control over the node that Elasticsearch will run on. The contents of DataNodeSelector will be added to the PodSpec of the Elasticsearch nodes. For the pod to be eligible to run on a node, the node must have each of the indicated key-value pairs as labels as well as access to the specified StorageClassName. | | `componentResources` *[LogStorageComponentResource](#logstoragecomponentresource) array* | (Optional) ComponentResources can be used to customize the resource requirements for each component. Only ECKOperator is supported for this spec. | | `eckOperatorStatefulSet` *[ECKOperatorStatefulSet](#eckoperatorstatefulset)* | (Optional) ECKOperatorStatefulSet configures the ECKOperator StatefulSet. If used in conjunction with the deprecated ComponentResources, then these overrides take precedence. | | `kibana` *[Kibana](#kibana)* | (Optional) Kibana configures the Kibana Spec. | | `linseedDeployment` *[LinseedDeployment](#linseeddeployment)* | LinseedDeployment configures the linseed Deployment. | | `elasticsearchMetricsDeployment` *[ElasticsearchMetricsDeployment](#elasticsearchmetricsdeployment)* | ElasticsearchMetricsDeployment configures the tigera-elasticsearch-metric Deployment. | | `esGatewayDeployment` *[ESGatewayDeployment](#esgatewaydeployment)* | ESGatewayDeployment configures the es-gateway Deployment. | ### LogStorageStatus[​](#logstoragestatus) LogStorageStatus defines the observed state of Tigera flow and DNS log storage. *Appears in:* - [LogStorage](#logstorage) | Field | Description | | ------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `state` *string* | State provides user-readable status. | | `elasticsearchHash` *string* | ElasticsearchHash represents the current revision and configuration of the installed Elasticsearch cluster. This is an opaque string which can be monitored for changes to perform actions when Elasticsearch is modified. | | `kibanaHash` *string* | KibanaHash represents the current revision and configuration of the installed Kibana dashboard. This is an opaque string which can be monitored for changes to perform actions when Kibana is modified. | | `conditions` *[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#condition-v1-meta) array* | (Optional) Conditions represents the latest observed set of conditions for the component. A component may be one or more of Ready, Progressing, Degraded or other customer types. | ### Logging[​](#logging) *Appears in:* - [InstallationSpec](#installationspec) | Field | Description | | --------------------------------- | ----------------------------------------------------------------- | | `cni` *[CNILogging](#cnilogging)* | (Optional) Customized logging specification for calico-cni plugin | ### ManagementCluster[​](#managementcluster) The presence of ManagementCluster in your cluster, will configure it to be the management plane to which managed clusters can connect. At most one instance of this resource is supported. It must be named "tigera-secure". | Field | Description | | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------- | | `apiVersion` *string* | `operator.tigera.io/v1` | | `kind` *string* | `ManagementCluster` | | `metadata` *[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#objectmeta-v1-meta)* | Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[ManagementClusterSpec](#managementclusterspec)* | | ### ManagementClusterConnection[​](#managementclusterconnection) ManagementClusterConnection represents a link between a managed cluster and a management cluster. At most one instance of this resource is supported. It must be named "tigera-secure". | Field | Description | | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------- | | `apiVersion` *string* | `operator.tigera.io/v1` | | `kind` *string* | `ManagementClusterConnection` | | `metadata` *[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#objectmeta-v1-meta)* | Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[ManagementClusterConnectionSpec](#managementclusterconnectionspec)* | | | `status` *[ManagementClusterConnectionStatus](#managementclusterconnectionstatus)* | | ### ManagementClusterConnectionSpec[​](#managementclusterconnectionspec) ManagementClusterConnectionSpec defines the desired state of ManagementClusterConnection *Appears in:* - [ManagementClusterConnection](#managementclusterconnection) | Field | Description | | ---------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `managementClusterAddr` *string* | (Optional) Specify where the managed cluster can reach the management cluster. Ex.: "10.128.0.10:30449". A managed cluster should be able to access this address. This field is used by managed clusters only. | | `tls` *[ManagementClusterTLS](#managementclustertls)* | (Optional) TLS provides options for configuring how Managed Clusters can establish an mTLS connection with the Management Cluster. | | `guardianDeployment` *[GuardianDeployment](#guardiandeployment)* | GuardianDeployment configures the guardian Deployment. | | `impersonation` *[Impersonation](#impersonation)* | (Optional) Impersonation configures the RBAC impersonation permissions for the guardian deployment. This field is not applicable to installation variant Calico as no impersonation is ever used. Otherwise, if this field is left nil, a default set of permissions will be applied. WARNING: If this field is specified, it completely replaces the default permissions. For example, providing an empty `impersonation: \{\}` block will result in guardian having NO impersonation permissions. Similarly, if you specify `users` but omit `groups`, guardian will lose its default permissions to impersonate groups. | ### ManagementClusterConnectionStatus[​](#managementclusterconnectionstatus) ManagementClusterConnectionStatus defines the observed state of ManagementClusterConnection *Appears in:* - [ManagementClusterConnection](#managementclusterconnection) | Field | Description | | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `conditions` *[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#condition-v1-meta) array* | (Optional) Conditions represents the latest observed set of conditions for the component. A component may be one or more of Ready, Progressing, Degraded or other customer types. | ### ManagementClusterSpec[​](#managementclusterspec) ManagementClusterSpec defines the desired state of a ManagementCluster *Appears in:* - [ManagementCluster](#managementcluster) | Field | Description | | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `address` *string* | (Optional) This field specifies the externally reachable address to which your managed cluster will connect. When a managed cluster is added, this field is used to populate an easy-to-apply manifest that will connect both clusters. Valid examples are: "0.0.0.0:31000", "example.com:32000", "\[::1]:32500" | | `tls` *[TLS](#tls)* | (Optional) TLS provides options for configuring how Managed Clusters can establish an mTLS connection with the Management Cluster. | ### ManagementClusterTLS[​](#managementclustertls) *Appears in:* - [ManagementClusterConnectionSpec](#managementclusterconnectionspec) | Field | Description | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ca` *[CAType](#catype)* | CA indicates which verification method the tunnel client should use to verify the tunnel server's identity. When left blank or set to 'Tigera', the tunnel client will expect a self-signed cert to be included in the certificate bundle and will expect the cert to have a Common Name (CN) of 'voltron'. When set to 'Public', the tunnel client will use its installed system certs and will use the managementClusterAddr to verify the tunnel server's identity. Default: Tigera | ### Manager[​](#manager) Manager installs the Calico Enterprise manager graphical user interface. At most one instance of this resource is supported. It must be named "tigera-secure". | Field | Description | | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------- | | `apiVersion` *string* | `operator.tigera.io/v1` | | `kind` *string* | `Manager` | | `metadata` *[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#objectmeta-v1-meta)* | Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[ManagerSpec](#managerspec)* | Specification of the desired state for the Calico Enterprise manager. | | `status` *[ManagerStatus](#managerstatus)* | Most recently observed state for the Calico Enterprise manager. | ### ManagerDeployment[​](#managerdeployment) ManagerDeployment is the configuration for the Manager Deployment. *Appears in:* - [ManagerSpec](#managerspec) | Field | Description | | -------------------------------------------------------- | --------------------------------------------------------------- | | `spec` *[ManagerDeploymentSpec](#managerdeploymentspec)* | (Optional) Spec is the specification of the Manager Deployment. | ### ManagerDeploymentContainer[​](#managerdeploymentcontainer) ManagerDeploymentContainer is a Manager Deployment container. *Appears in:* - [ManagerDeploymentPodSpec](#managerdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the Manager Deployment container by name. Supported values are: tigera-voltron, tigera-manager, tigera-ui-apis, and tigera-es-proxy (deprecated). | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named Manager Deployment container's resources. If omitted, the Manager Deployment will use its default value for this container's resources. | ### ManagerDeploymentInitContainer[​](#managerdeploymentinitcontainer) ManagerDeploymentInitContainer is a Manager Deployment init container. *Appears in:* - [ManagerDeploymentPodSpec](#managerdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the Manager Deployment init container by name. Supported values are: manager-tls-key-cert-provisioner, internal-manager-tls-key-cert-provisioner, tigera-voltron-linseed-tls-key-cert-provisioner | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named Manager Deployment init container's resources. If omitted, the Manager Deployment will use its default value for this init container's resources. If used in conjunction with the deprecated ComponentResources, then this value takes precedence. | ### ManagerDeploymentPodSpec[​](#managerdeploymentpodspec) ManagerDeploymentPodSpec is the Manager Deployment's PodSpec. *Appears in:* - [ManagerDeploymentPodTemplateSpec](#managerdeploymentpodtemplatespec) | Field | Description | | ------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[ManagerDeploymentInitContainer](#managerdeploymentinitcontainer) array* | (Optional) InitContainers is a list of Manager init containers. If specified, this overrides the specified Manager Deployment init containers. If omitted, the Manager Deployment will use its default values for its init containers. | | `containers` *[ManagerDeploymentContainer](#managerdeploymentcontainer) array* | (Optional) Containers is a list of Manager containers. If specified, this overrides the specified Manager Deployment containers. If omitted, the Manager Deployment will use its default values for its containers. | ### ManagerDeploymentPodTemplateSpec[​](#managerdeploymentpodtemplatespec) ManagerDeploymentPodTemplateSpec is the Manager Deployment's PodTemplateSpec *Appears in:* - [ManagerDeploymentSpec](#managerdeploymentspec) | Field | Description | | -------------------------------------------------------------- | ---------------------------------------------------- | | `spec` *[ManagerDeploymentPodSpec](#managerdeploymentpodspec)* | (Optional) Spec is the Manager Deployment's PodSpec. | ### ManagerDeploymentSpec[​](#managerdeploymentspec) ManagerDeploymentSpec defines configuration for the Manager Deployment. *Appears in:* - [ManagerDeployment](#managerdeployment) | Field | Description | | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | | `template` *[ManagerDeploymentPodTemplateSpec](#managerdeploymentpodtemplatespec)* | (Optional) Template describes the Manager Deployment pod that will be created. | ### ManagerSpec[​](#managerspec) ManagerSpec defines configuration for the Calico Enterprise manager GUI. *Appears in:* - [Manager](#manager) | Field | Description | | ------------------------------------------------------------- | --------------------------------------------------------------- | | `managerDeployment` *[ManagerDeployment](#managerdeployment)* | (Optional) ManagerDeployment configures the Manager Deployment. | ### ManagerStatus[​](#managerstatus) ManagerStatus defines the observed state of the Calico Enterprise manager GUI. *Appears in:* - [Manager](#manager) | Field | Description | | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `state` *string* | State provides user-readable status. | | `conditions` *[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#condition-v1-meta) array* | (Optional) Conditions represents the latest observed set of conditions for the component. A component may be one or more of Ready, Progressing, Degraded or other customer types. | ### Metadata[​](#metadata) Metadata contains the standard Kubernetes labels and annotations fields. *Appears in:* - [APIServerDeployment](#apiserverdeployment) - [APIServerDeploymentPodTemplateSpec](#apiserverdeploymentpodtemplatespec) - [CSINodeDriverDaemonSet](#csinodedriverdaemonset) - [CSINodeDriverDaemonSetPodTemplateSpec](#csinodedriverdaemonsetpodtemplatespec) - [CalicoKubeControllersDeployment](#calicokubecontrollersdeployment) - [CalicoKubeControllersDeploymentPodTemplateSpec](#calicokubecontrollersdeploymentpodtemplatespec) - [CalicoNodeDaemonSet](#caliconodedaemonset) - [CalicoNodeDaemonSetPodTemplateSpec](#caliconodedaemonsetpodtemplatespec) - [CalicoNodeWindowsDaemonSet](#caliconodewindowsdaemonset) - [CalicoNodeWindowsDaemonSetPodTemplateSpec](#caliconodewindowsdaemonsetpodtemplatespec) - [CalicoWindowsUpgradeDaemonSet](#calicowindowsupgradedaemonset) - [CalicoWindowsUpgradeDaemonSetPodTemplateSpec](#calicowindowsupgradedaemonsetpodtemplatespec) - [GatewayCertgenJob](#gatewaycertgenjob) - [GatewayCertgenJobPodTemplate](#gatewaycertgenjobpodtemplate) - [GatewayControllerDeployment](#gatewaycontrollerdeployment) - [GatewayControllerDeploymentPodTemplate](#gatewaycontrollerdeploymentpodtemplate) - [GatewayDaemonSetPodTemplate](#gatewaydaemonsetpodtemplate) - [GatewayDeploymentPodTemplate](#gatewaydeploymentpodtemplate) - [GatewayService](#gatewayservice) - [GoldmaneDeployment](#goldmanedeployment) - [GoldmaneDeploymentPodTemplateSpec](#goldmanedeploymentpodtemplatespec) - [TyphaDeployment](#typhadeployment) - [TyphaDeploymentPodTemplateSpec](#typhadeploymentpodtemplatespec) - [WhiskerDeployment](#whiskerdeployment) - [WhiskerDeploymentPodTemplateSpec](#whiskerdeploymentpodtemplatespec) | Field | Description | | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `labels` *object (keys:string, values:string)* | (Optional) Labels is a map of string keys and values that may match replicaset and service selectors. Each of these key/value pairs are added to the object's labels provided the key does not already exist in the object's labels. | | `annotations` *object (keys:string, values:string)* | (Optional) Annotations is a map of arbitrary non-identifying metadata. Each of these key/value pairs are added to the object's annotations provided the key does not already exist in the object's annotations. | ### Monitor[​](#monitor) Monitor is the Schema for the monitor API. At most one instance of this resource is supported. It must be named "tigera-secure". | Field | Description | | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------- | | `apiVersion` *string* | `operator.tigera.io/v1` | | `kind` *string* | `Monitor` | | `metadata` *[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#objectmeta-v1-meta)* | Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[MonitorSpec](#monitorspec)* | | | `status` *[MonitorStatus](#monitorstatus)* | | ### MonitorSpec[​](#monitorspec) MonitorSpec defines the desired state of Tigera monitor. *Appears in:* - [Monitor](#monitor) | Field | Description | | ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `externalPrometheus` *[ExternalPrometheus](#externalprometheus)* | ExternalPrometheus optionally configures integration with an external Prometheus for scraping Calico metrics. When specified, the operator will render resources in the defined namespace. This option can be useful for configuring scraping from git-ops tools without the need of post-installation steps. | | `prometheus` *[Prometheus](#prometheus)* | (Optional) Prometheus is the configuration for the Prometheus. | | `alertManager` *[AlertManager](#alertmanager)* | (Optional) AlertManager is the configuration for the AlertManager. | ### MonitorStatus[​](#monitorstatus) MonitorStatus defines the observed state of Tigera monitor. *Appears in:* - [Monitor](#monitor) | Field | Description | | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `state` *string* | State provides user-readable status. | | `conditions` *[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#condition-v1-meta) array* | (Optional) Conditions represents the latest observed set of conditions for the component. A component may be one or more of Ready, Progressing, Degraded or other customer types. | ### MultiInterfaceMode[​](#multiinterfacemode) *Underlying type:* *string* MultiInterfaceMode describes the method of providing multiple pod interfaces. One of: None, Multus *Appears in:* - [CalicoNetworkSpec](#caliconetworkspec) | Value | Description | | -------- | ----------- | | `None` | | | `Multus` | | ### NATOutgoingType[​](#natoutgoingtype) *Underlying type:* *string* NATOutgoingType describe the type of outgoing NAT to use. One of: Enabled, Disabled *Appears in:* - [IPPool](#ippool) | Value | Description | | ---------- | ----------- | | `Enabled` | | | `Disabled` | | ### NamespacedName[​](#namespacedname) NamespacedName references an object of a known type in any namespace. *Appears in:* - [GatewayAPISpec](#gatewayapispec) - [GatewayClassSpec](#gatewayclassspec) | Field | Description | | -------------------- | ----------- | | `namespace` *string* | | | `name` *string* | | ### NativeIP[​](#nativeip) *Underlying type:* *string* NativeIP defines if Egress Gateway pods should have AWS IPs. When NativeIP is enabled, the IPPools should be backed by AWS subnet. *Appears in:* - [AWSEgressGateway](#awsegressgateway) | Value | Description | | ---------- | ----------- | | `Enabled` | | | `Disabled` | | ### NodeAddressAutodetection[​](#nodeaddressautodetection) NodeAddressAutodetection provides configuration options for auto-detecting node addresses. At most one option can be used. If no detection option is specified, then IP auto detection will be disabled for this address family and IPs must be specified directly on the Node resource. *Appears in:* - [CalicoNetworkSpec](#caliconetworkspec) | Field | Description | | ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `firstFound` *boolean* | (Optional) FirstFound uses default interface matching parameters to select an interface, performing best-effort filtering based on well-known interface names. | | `kubernetes` *[KubernetesAutodetectionMethod](#kubernetesautodetectionmethod)* | (Optional) Kubernetes configures Calico to detect node addresses based on the Kubernetes API. | | `interface` *string* | (Optional) Interface enables IP auto-detection based on interfaces that match the given regex. | | `skipInterface` *string* | (Optional) SkipInterface enables IP auto-detection based on interfaces that do not match the given regex. | | `canReach` *string* | (Optional) CanReach enables IP auto-detection based on which source address on the node is used to reach the specified IP or domain. | | `cidrs` *string array* | CIDRS enables IP auto-detection based on which addresses on the nodes are within one of the provided CIDRs. | ### NodeAffinity[​](#nodeaffinity) NodeAffinity is similar to \*v1.NodeAffinity, but allows us to limit available schedulers. *Appears in:* - [TyphaAffinity](#typhaaffinity) | Field | Description | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `preferredDuringSchedulingIgnoredDuringExecution` *[PreferredSchedulingTerm](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#preferredschedulingterm-v1-core) array* | (Optional) The scheduler will prefer to schedule pods to nodes that satisfy the affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. | | `requiredDuringSchedulingIgnoredDuringExecution` *[NodeSelector](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#nodeselector-v1-core)* | (Optional) WARNING: Please note that if the affinity requirements specified by this field are not met at scheduling time, the pod will NOT be scheduled onto the node. There is no fallback to another affinity rules with this setting. This may cause networking disruption or even catastrophic failure! PreferredDuringSchedulingIgnoredDuringExecution should be used for affinity unless there is a specific well understood reason to use RequiredDuringSchedulingIgnoredDuringExecution and you can guarantee that the RequiredDuringSchedulingIgnoredDuringExecution will always have sufficient nodes to satisfy the requirement. NOTE: RequiredDuringSchedulingIgnoredDuringExecution is set by default for AKS nodes, to avoid scheduling Typhas on virtual-nodes. If the affinity requirements specified by this field cease to be met at some point during pod execution (e.g. due to an update), the system may or may not try to eventually evict the pod from its node. | ### NodeSet[​](#nodeset) NodeSets defines configuration specific to each Elasticsearch Node Set *Appears in:* - [Nodes](#nodes) | Field | Description | | ------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `selectionAttributes` *[NodeSetSelectionAttribute](#nodesetselectionattribute) array* | SelectionAttributes defines K8s node attributes a NodeSet should use when setting the Node Affinity selectors and Elasticsearch cluster awareness attributes for the Elasticsearch nodes. The list of SelectionAttributes are used to define Node Affinities and set the node awareness configuration in the running Elasticsearch instance. | ### NodeSetSelectionAttribute[​](#nodesetselectionattribute) NodeSetSelectionAttribute defines a K8s node "attribute" the Elasticsearch nodes should be aware of. The "Name" and "Value" are used together to set the "awareness" attributes in Elasticsearch, while the "NodeLabel" and "Value" are used together to define Node Affinity for the Pods created for the Elasticsearch nodes. *Appears in:* - [NodeSet](#nodeset) | Field | Description | | -------------------- | ----------- | | `name` *string* | | | `nodeLabel` *string* | | | `value` *string* | | ### Nodes[​](#nodes) Nodes defines the configuration for a set of identical Elasticsearch cluster nodes, each of type master, data, and ingest. *Appears in:* - [LogStorageSpec](#logstoragespec) | Field | Description | | -------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | `count` *integer* | Count defines the number of nodes in the Elasticsearch cluster. | | `nodeSets` *[NodeSet](#nodeset) array* | (Optional) NodeSets defines configuration specific to each Elasticsearch Node Set | | `resourceRequirements` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) ResourceRequirements defines the resource limits and requirements for the Elasticsearch cluster. | ### NonClusterHost[​](#nonclusterhost) NonClusterHost installs the components required for non-cluster host log collection. At most one instance of this resource is supported. It must be named "tigera-secure". | Field | Description | | ------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------- | | `apiVersion` *string* | `operator.tigera.io/v1` | | `kind` *string* | `NonClusterHost` | | `metadata` *[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#objectmeta-v1-meta)* | Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[NonClusterHostSpec](#nonclusterhostspec)* | Specification of the desired state for non-cluster host log collection. | ### NonClusterHostSpec[​](#nonclusterhostspec) NonClusterHostSpec enables non-cluster hosts to connect to a cluster. *Appears in:* - [NonClusterHost](#nonclusterhost) | Field | Description | | ------------------------ | ------------------------------------------------------------------------------------------------------------------ | | `endpoint` *string* | Location of the log ingestion point for non-cluster hosts. For example: [https://1.2.3.4:443](https://1.2.3.4:443) | | `typhaEndpoint` *string* | Location of the Typha endpoint for non-cluster host Felix and Typha communication. For example: 5.6.7.8:5473 | ### NonPrivilegedType[​](#nonprivilegedtype) *Underlying type:* *string* NonPrivilegedType specifies whether Calico runs as permissioned or not One of: Enabled, Disabled *Appears in:* - [InstallationSpec](#installationspec) | Value | Description | | ---------- | ----------- | | `Enabled` | | | `Disabled` | | ### NotificationMode[​](#notificationmode) *Underlying type:* *string* *Appears in:* - [WhiskerSpec](#whiskerspec) | Value | Description | | ---------- | ----------- | | `Disabled` | | | `Enabled` | | ### OIDCType[​](#oidctype) *Underlying type:* *string* OIDCType defines how OIDC is configured for Tigera Enterprise. Dex should be the best option for most use-cases. The Tigera option can help in specific use-cases, for instance, when you are unable to configure a client secret. One of: Dex, Tigera *Validation:* - Enum: \[Dex Tigera] *Appears in:* - [AuthenticationOIDC](#authenticationoidc) | Value | Description | | -------- | --------------------------------------------------------------------------------------------------- | | `Dex` | OIDCTypeDex uses Dex IdP, a popular open-source tool for connecting OIDC. | | `Tigera` | OIDCTypeTigera uses customer code to pass OIDC configuration directly into our server applications. | ### PacketCaptureAPI[​](#packetcaptureapi) PacketCaptureAPI is used to configure the resource requirement for PacketCaptureAPI deployment. It must be named "tigera-secure". | Field | Description | | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------- | | `apiVersion` *string* | `operator.tigera.io/v1` | | `kind` *string* | `PacketCaptureAPI` | | `metadata` *[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#objectmeta-v1-meta)* | Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[PacketCaptureAPISpec](#packetcaptureapispec)* | Specification of the desired state for the PacketCaptureAPI. | | `status` *[PacketCaptureAPIStatus](#packetcaptureapistatus)* | Most recently observed state for the PacketCaptureAPI. | ### PacketCaptureAPIDeployment[​](#packetcaptureapideployment) PacketCaptureAPIDeployment is the configuration for the PacketCaptureAPI Deployment. *Appears in:* - [PacketCaptureAPISpec](#packetcaptureapispec) | Field | Description | | -------------------------------------------------------------------------- | ------------------------------------------------------------------------ | | `spec` *[PacketCaptureAPIDeploymentSpec](#packetcaptureapideploymentspec)* | (Optional) Spec is the specification of the PacketCaptureAPI Deployment. | ### PacketCaptureAPIDeploymentContainer[​](#packetcaptureapideploymentcontainer) PacketCaptureAPIDeploymentContainer is a PacketCaptureAPI Deployment container. *Appears in:* - [PacketCaptureAPIDeploymentPodSpec](#packetcaptureapideploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the PacketCaptureAPI Deployment container by name. Supported values are: tigera-packetcapture-server | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named PacketCaptureAPI Deployment container's resources. If omitted, the PacketCaptureAPI Deployment will use its default value for this container's resources. | ### PacketCaptureAPIDeploymentInitContainer[​](#packetcaptureapideploymentinitcontainer) PacketCaptureAPIDeploymentInitContainer is a PacketCaptureAPI Deployment init container. *Appears in:* - [PacketCaptureAPIDeploymentPodSpec](#packetcaptureapideploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the PacketCaptureAPI Deployment init container by name. Supported values are: tigera-packetcapture-server-tls-key-cert-provisioner | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named PacketCaptureAPI Deployment init container's resources. If omitted, the PacketCaptureAPI Deployment will use its default value for this init container's resources. | ### PacketCaptureAPIDeploymentPodSpec[​](#packetcaptureapideploymentpodspec) PacketCaptureAPIDeploymentPodSpec is the PacketCaptureAPI Deployment's PodSpec. *Appears in:* - [PacketCaptureAPIDeploymentPodTemplateSpec](#packetcaptureapideploymentpodtemplatespec) | Field | Description | | ------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[PacketCaptureAPIDeploymentInitContainer](#packetcaptureapideploymentinitcontainer) array* | (Optional) InitContainers is a list of PacketCaptureAPI init containers. If specified, this overrides the specified PacketCaptureAPI Deployment init containers. If omitted, the PacketCaptureAPI Deployment will use its default values for its init containers. | | `containers` *[PacketCaptureAPIDeploymentContainer](#packetcaptureapideploymentcontainer) array* | (Optional) Containers is a list of PacketCaptureAPI containers. If specified, this overrides the specified PacketCaptureAPI Deployment containers. If omitted, the PacketCaptureAPI Deployment will use its default values for its containers. | ### PacketCaptureAPIDeploymentPodTemplateSpec[​](#packetcaptureapideploymentpodtemplatespec) PacketCaptureAPIDeploymentPodTemplateSpec is the PacketCaptureAPI Deployment's PodTemplateSpec *Appears in:* - [PacketCaptureAPIDeploymentSpec](#packetcaptureapideploymentspec) | Field | Description | | -------------------------------------------------------------------------------- | ------------------------------------------------------------- | | `spec` *[PacketCaptureAPIDeploymentPodSpec](#packetcaptureapideploymentpodspec)* | (Optional) Spec is the PacketCaptureAPI Deployment's PodSpec. | ### PacketCaptureAPIDeploymentSpec[​](#packetcaptureapideploymentspec) PacketCaptureAPIDeploymentSpec defines configuration for the PacketCaptureAPI Deployment. *Appears in:* - [PacketCaptureAPIDeployment](#packetcaptureapideployment) | Field | Description | | ---------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | `template` *[PacketCaptureAPIDeploymentPodTemplateSpec](#packetcaptureapideploymentpodtemplatespec)* | (Optional) Template describes the PacketCaptureAPI Deployment pod that will be created. | ### PacketCaptureAPISpec[​](#packetcaptureapispec) PacketCaptureAPISpec defines configuration for the Packet Capture API. *Appears in:* - [PacketCaptureAPI](#packetcaptureapi) | Field | Description | | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | | `packetCaptureAPIDeployment` *[PacketCaptureAPIDeployment](#packetcaptureapideployment)* | (Optional) PacketCaptureAPIDeployment configures the PacketCaptureAPI Deployment. | ### PacketCaptureAPIStatus[​](#packetcaptureapistatus) PacketCaptureAPIStatus defines the observed state of the Packet Capture API. *Appears in:* - [PacketCaptureAPI](#packetcaptureapi) | Field | Description | | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `state` *string* | State provides user-readable status. | | `conditions` *[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#condition-v1-meta) array* | (Optional) Conditions represents the latest observed set of conditions for the component. A component may be one or more of Ready, Progressing, Degraded or other customer types. | ### PathMatch[​](#pathmatch) *Appears in:* - [TLSTerminatedRouteSpec](#tlsterminatedroutespec) | Field | Description | | ---------------------- | ----------------------------------------------------------------------------- | | `path` *string* | Path is the path portion of the URL based on which we proxy. | | `pathRegexp` *string* | (Optional) PathRegexp, if not nil, checks if Regexp matches the path. | | `pathReplace` *string* | (Optional) PathReplace if not nil will be used to replace PathRegexp matches. | ### PolicyMode[​](#policymode) *Underlying type:* *string* *Appears in:* - [Azure](#azure) | Value | Description | | --------- | ----------- | | `Default` | | | `Manual` | | ### PolicyRecommendation[​](#policyrecommendation) PolicyRecommendation is the Schema for the policy recommendation API. At most one instance of this resource is supported. It must be named "tigera-secure". | Field | Description | | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------- | | `apiVersion` *string* | `operator.tigera.io/v1` | | `kind` *string* | `PolicyRecommendation` | | `metadata` *[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#objectmeta-v1-meta)* | Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[PolicyRecommendationSpec](#policyrecommendationspec)* | | | `status` *[PolicyRecommendationStatus](#policyrecommendationstatus)* | | ### PolicyRecommendationDeployment[​](#policyrecommendationdeployment) PolicyRecommendationDeployment is the configuration for the PolicyRecommendation Deployment. *Appears in:* - [PolicyRecommendationSpec](#policyrecommendationspec) | Field | Description | | ---------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | | `spec` *[PolicyRecommendationDeploymentSpec](#policyrecommendationdeploymentspec)* | (Optional) Spec is the specification of the PolicyRecommendation Deployment. | ### PolicyRecommendationDeploymentContainer[​](#policyrecommendationdeploymentcontainer) PolicyRecommendationDeploymentContainer is a PolicyRecommendation Deployment container. *Appears in:* - [PolicyRecommendationDeploymentPodSpec](#policyrecommendationdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the PolicyRecommendation Deployment container by name. Supported values are: policy-recommendation-controller | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named PolicyRecommendation Deployment container's resources. If omitted, the PolicyRecommendation Deployment will use its default value for this container's resources. | ### PolicyRecommendationDeploymentInitContainer[​](#policyrecommendationdeploymentinitcontainer) PolicyRecommendationDeploymentInitContainer is a PolicyRecommendation Deployment init container. *Appears in:* - [PolicyRecommendationDeploymentPodSpec](#policyrecommendationdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the PolicyRecommendation Deployment init container by name. | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named PolicyRecommendation Deployment init container's resources. If omitted, the PolicyRecommendation Deployment will use its default value for this init container's resources. | ### PolicyRecommendationDeploymentPodSpec[​](#policyrecommendationdeploymentpodspec) PolicyRecommendationDeploymentPodSpec is the PolicyRecommendation Deployment's PodSpec. *Appears in:* - [PolicyRecommendationDeploymentPodTemplateSpec](#policyrecommendationdeploymentpodtemplatespec) | Field | Description | | -------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[PolicyRecommendationDeploymentInitContainer](#policyrecommendationdeploymentinitcontainer) array* | (Optional) InitContainers is a list of PolicyRecommendation init containers. If specified, this overrides the specified PolicyRecommendation Deployment init containers. If omitted, the PolicyRecommendation Deployment will use its default values for its init containers. | | `containers` *[PolicyRecommendationDeploymentContainer](#policyrecommendationdeploymentcontainer) array* | (Optional) Containers is a list of PolicyRecommendation containers. If specified, this overrides the specified PolicyRecommendation Deployment containers. If omitted, the PolicyRecommendation Deployment will use its default values for its containers. | ### PolicyRecommendationDeploymentPodTemplateSpec[​](#policyrecommendationdeploymentpodtemplatespec) PolicyRecommendationDeploymentPodTemplateSpec is the PolicyRecommendation Deployment's PodTemplateSpec *Appears in:* - [PolicyRecommendationDeploymentSpec](#policyrecommendationdeploymentspec) | Field | Description | | ---------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | | `spec` *[PolicyRecommendationDeploymentPodSpec](#policyrecommendationdeploymentpodspec)* | (Optional) Spec is the PolicyRecommendation Deployment's PodSpec. | ### PolicyRecommendationDeploymentSpec[​](#policyrecommendationdeploymentspec) PolicyRecommendationDeploymentSpec defines configuration for the PolicyRecommendation Deployment. *Appears in:* - [PolicyRecommendationDeployment](#policyrecommendationdeployment) | Field | Description | | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------- | | `template` *[PolicyRecommendationDeploymentPodTemplateSpec](#policyrecommendationdeploymentpodtemplatespec)* | (Optional) Template describes the PolicyRecommendation Deployment pod that will be created. | ### PolicyRecommendationSpec[​](#policyrecommendationspec) PolicyRecommendationSpec defines configuration for the Calico Enterprise Policy Recommendation service. *Appears in:* - [PolicyRecommendation](#policyrecommendation) | Field | Description | | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | | `policyRecommendationDeployment` *[PolicyRecommendationDeployment](#policyrecommendationdeployment)* | (Optional) PolicyRecommendation configures the PolicyRecommendation Deployment. | ### PolicyRecommendationStatus[​](#policyrecommendationstatus) PolicyRecommendationStatus defines the observed state of Tigera policy recommendation. *Appears in:* - [PolicyRecommendation](#policyrecommendation) | Field | Description | | ---------------- | ------------------------------------ | | `state` *string* | State provides user-readable status. | ### ProductVariant[​](#productvariant) *Underlying type:* *string* ProductVariant represents the variant of the product. One of: Calico, TigeraSecureEnterprise *Appears in:* - [InstallationSpec](#installationspec) - [InstallationStatus](#installationstatus) ### Prometheus[​](#prometheus) *Appears in:* - [MonitorSpec](#monitorspec) | Field | Description | | ------------------------------------------ | ------------------------------------------------------- | | `spec` *[PrometheusSpec](#prometheusspec)* | (Optional) Spec is the specification of the Prometheus. | ### PrometheusContainer[​](#prometheuscontainer) PrometheusContainer is a Prometheus container. *Appears in:* - [CommonPrometheusFields](#commonprometheusfields) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the Prometheus Deployment container by name. Supported values are: authn-proxy | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named Prometheus container's resources. If omitted, the Prometheus will use its default value for this container's resources. | ### PrometheusSpec[​](#prometheusspec) *Appears in:* - [Prometheus](#prometheus) | Field | Description | | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | `commonPrometheusFields` *[CommonPrometheusFields](#commonprometheusfields)* | CommonPrometheusFields are the options available to both the Prometheus server and agent. | ### PromptType[​](#prompttype) *Underlying type:* *string* PromptType is a value that specifies whether the identity provider prompts the end user for re-authentication and consent. One of: None, Login, Consent, SelectAccount. *Validation:* - Enum: \[None Login Consent SelectAccount] *Appears in:* - [AuthenticationOIDC](#authenticationoidc) | Value | Description | | --------------- | -------------------------------------------------------------------------------------------------------- | | `None` | The identity provider must not display any authentication or consent user interface pages. | | `Login` | The identity provider should prompt the end user for reauthentication. | | `Consent` | The identity provider should prompt the end user for consent before returning information to the client. | | `SelectAccount` | The identity provider should prompt the end user to select a user account. | ### Provider[​](#provider) *Underlying type:* *string* Provider represents a particular provider or flavor of Kubernetes. Valid options are: EKS, GKE, AKS, RKE2, OpenShift, DockerEnterprise, TKG, Kind. *Appears in:* - [InstallationSpec](#installationspec) ### Proxy[​](#proxy) *Appears in:* - [InstallationSpec](#installationspec) | Field | Description | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `httpProxy` *string* | (Optional) HTTPProxy defines the value of the HTTP\_PROXY environment variable that will be set on Tigera containers that connect to destinations outside the cluster. | | `httpsProxy` *string* | (Optional) HTTPSProxy defines the value of the HTTPS\_PROXY environment variable that will be set on Tigera containers that connect to destinations outside the cluster. | | `noProxy` *string* | (Optional) NoProxy defines the value of the NO\_PROXY environment variable that will be set on Tigera containers that connect to destinations outside the cluster. This value must be set such that destinations within the scope of the cluster, including the Kubernetes API server, are exempt from being proxied. | ### QueryServerLogging[​](#queryserverlogging) *Appears in:* - [APIServerPodLogging](#apiserverpodlogging) | Field | Description | | ------------------------------------------- | ------------------------------------------------------------------- | | `logSeverity` *[LogSeverity](#logseverity)* | (Optional) LogSeverity defines log level for QueryServer container. | ### Retention[​](#retention) Retention defines how long data is retained in an Elasticsearch cluster before it is cleared. *Appears in:* - [LogStorageSpec](#logstoragespec) | Field | Description | | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `flows` *integer* | (Optional) Flows configures the retention period for flow logs, in days. Logs written on a day that started at least this long ago are removed. To keep logs for at least x days, use a retention period of x+1. Default: 8 | | `auditReports` *integer* | (Optional) AuditReports configures the retention period for audit logs, in days. Logs written on a day that started at least this long ago are removed. To keep logs for at least x days, use a retention period of x+1. Default: 91 | | `snapshots` *integer* | (Optional) Snapshots configures the retention period for snapshots, in days. Snapshots are periodic captures of resources which along with audit events are used to generate reports. Consult the Compliance Reporting documentation for more details on snapshots. Logs written on a day that started at least this long ago are removed. To keep logs for at least x days, use a retention period of x+1. Default: 91 | | `complianceReports` *integer* | (Optional) ComplianceReports configures the retention period for compliance reports, in days. Reports are output from the analysis of the system state and audit events for compliance reporting. Consult the Compliance Reporting documentation for more details on reports. Logs written on a day that started at least this long ago are removed. To keep logs for at least x days, use a retention period of x+1. Default: 91 | | `dnsLogs` *integer* | (Optional) DNSLogs configures the retention period for DNS logs, in days. Logs written on a day that started at least this long ago are removed. To keep logs for at least x days, use a retention period of x+1. Default: 8 | | `bgpLogs` *integer* | (Optional) BGPLogs configures the retention period for BGP logs, in days. Logs written on a day that started at least this long ago are removed. To keep logs for at least x days, use a retention period of x+1. Default: 8 | ### S3StoreSpec[​](#s3storespec) S3StoreSpec defines configuration for exporting logs to Amazon S3. *Appears in:* - [AdditionalLogStoreSpec](#additionallogstorespec) | Field | Description | | ------------------------------------- | ----------------------------------------------------------------------- | | `region` *string* | AWS Region of the S3 bucket | | `bucketName` *string* | Name of the S3 bucket to send logs | | `bucketPath` *string* | Path in the S3 bucket where to send logs | | `hostScope` *[HostScope](#hostscope)* | (Optional) The set of hosts that will forward their logs to this store. | ### SNIMatch[​](#snimatch) *Appears in:* - [TLSPassThroughRouteSpec](#tlspassthroughroutespec) | Field | Description | | --------------------- | ------------------------------------------------------------ | | `serverName` *string* | ServerName is used to match the server name for the request. | ### ServiceMonitor[​](#servicemonitor) *Appears in:* - [ExternalPrometheus](#externalprometheus) | Field | Description | | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `labels` *object (keys:string, values:string)* | Labels are the metadata.labels of the ServiceMonitor. When combined with spec.serviceMonitorSelector.matchLabels on your prometheus instance, the service monitor will automatically be picked up. Default: k8s-app=tigera-prometheus | | `endpoints` *[Endpoint](#endpoint) array* | The endpoints to scrape. This struct contains a subset of the Endpoint as defined in the prometheus docs. Fields related to connecting to our Prometheus server are automatically set by the operator. | ### SidecarStatusType[​](#sidecarstatustype) *Underlying type:* *string* *Validation:* - Enum: \[Enabled Disabled] *Appears in:* - [ApplicationLayerSpec](#applicationlayerspec) | Value | Description | | ---------- | ----------- | | `Enabled` | | | `Disabled` | | ### SidecarWebhookStateType[​](#sidecarwebhookstatetype) *Underlying type:* *string* *Validation:* - Enum: \[Enabled Disabled] *Appears in:* - [ApplicationLayerStatus](#applicationlayerstatus) | Value | Description | | ---------- | ----------- | | `Enabled` | | | `Disabled` | | ### SplunkStoreSpec[​](#splunkstorespec) SplunkStoreSpec defines configuration for exporting logs to splunk. *Appears in:* - [AdditionalLogStoreSpec](#additionallogstorespec) | Field | Description | | ------------------------------------- | ------------------------------------------------------------------------------------ | | `endpoint` *string* | Location for splunk's http event collector end point. example `https://1.2.3.4:8088` | | `hostScope` *[HostScope](#hostscope)* | (Optional) The set of hosts that will forward their logs to this store | ### StatusConditionType[​](#statusconditiontype) *Underlying type:* *string* StatusConditionType is a type of condition that may apply to a particular component. *Appears in:* - [TigeraStatusCondition](#tigerastatuscondition) | Value | Description | | ------------- | --------------------------------------------------------------------------------------------------------------------------- | | `Available` | Available indicates that the component is healthy. | | `Progressing` | Progressing means that the component is in the process of being installed or upgraded. | | `Degraded` | Degraded means the component is not operating as desired and user action is required. | | `Ready` | Ready indicates that the component is healthy and ready.it is identical to Available and used in Status conditions for CRs. | ### Sysctl[​](#sysctl) *Appears in:* - [CalicoNetworkSpec](#caliconetworkspec) | Field | Description | | ---------------- | ----------- | | `key` *string* | | | `value` *string* | | ### SyslogLogType[​](#sysloglogtype) *Underlying type:* *string* SyslogLogType represents the allowable log types for syslog. Allowable values are Audit, DNS, Flows and IDSEvents. - Audit corresponds to audit logs for both Kubernetes resources and Enterprise custom resources. - DNS corresponds to DNS logs generated by Calico node. - Flows corresponds to flow logs generated by Calico node. - IDSEvents corresponds to event logs for the intrusion detection system (anomaly detection, suspicious IPs, suspicious domains and global alerts). *Validation:* - Enum: \[Audit DNS Flows IDSEvents] *Appears in:* - [SyslogStoreSpec](#syslogstorespec) | Value | Description | | ----------- | ----------- | | `Audit` | | | `DNS` | | | `Flows` | | | `L7` | | | `IDSEvents` | | ### SyslogStoreSpec[​](#syslogstorespec) SyslogStoreSpec defines configuration for exporting logs to syslog. *Appears in:* - [AdditionalLogStoreSpec](#additionallogstorespec) | Field | Description | | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `endpoint` *string* | Location of the syslog server. example: tcp\://1.2.3.4:601 | | `packetSize` *integer* | (Optional) PacketSize defines the maximum size of packets to send to syslog. In general this is only needed if you notice long logs being truncated. Default: 1024 | | `logTypes` *[SyslogLogType](#sysloglogtype) array* | If no values are provided, the list will be updated to include log types Audit, DNS and Flows. Default: Audit, DNS, Flows | | `encryption` *[EncryptionOption](#encryptionoption)* | (Optional) Encryption configures traffic encryption to the Syslog server. Default: None | | `hostScope` *[HostScope](#hostscope)* | (Optional) The set of hosts that will forward their logs to this store. | ### TLS[​](#tls) *Appears in:* - [ManagementClusterSpec](#managementclusterspec) | Field | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `secretName` *string* | (Optional) SecretName indicates the name of the secret in the tigera-operator namespace that contains the private key and certificate that the management cluster uses when it listens for incoming connections. When set to tigera-management-cluster-connection voltron will use the same cert bundle which Guardian client certs are signed with. When set to manager-tls, voltron will use the same cert bundle which Manager UI is served with. This cert bundle must be a publicly signed cert created by the user. Note that Tigera Operator will generate a self-signed manager-tls cert if one does not exist, and use of that cert will result in Guardian being unable to verify Voltron's identity. If changed on a running cluster with connected managed clusters, all managed clusters will disconnect as they will no longer be able to verify Voltron's identity. To reconnect existing managed clusters, change the tls.ca of the managed clusters' ManagementClusterConnection resource. One of: tigera-management-cluster-connection, manager-tls Default: tigera-management-cluster-connection | ### TLSCipher[​](#tlscipher) *Underlying type:* *string* *Validation:* - Enum: \[TLS\_AES\_256\_GCM\_SHA384 TLS\_CHACHA20\_POLY1305\_SHA256 TLS\_AES\_128\_GCM\_SHA256 TLS\_ECDHE\_ECDSA\_WITH\_AES\_256\_GCM\_SHA384 TLS\_ECDHE\_RSA\_WITH\_AES\_256\_GCM\_SHA384 TLS\_ECDHE\_RSA\_WITH\_CHACHA20\_POLY1305\_SHA256 TLS\_ECDHE\_ECDSA\_WITH\_CHACHA20\_POLY1305\_SHA256 TLS\_ECDHE\_RSA\_WITH\_AES\_128\_GCM\_SHA256 TLS\_ECDHE\_ECDSA\_WITH\_AES\_128\_GCM\_SHA256 TLS\_RSA\_WITH\_AES\_256\_GCM\_SHA384 TLS\_RSA\_WITH\_AES\_128\_GCM\_SHA256 TLS\_ECDHE\_ECDSA\_WITH\_AES\_256\_CBC\_SHA TLS\_ECDHE\_RSA\_WITH\_AES\_256\_CBC\_SHA TLS\_ECDHE\_RSA\_WITH\_AES\_128\_CBC\_SHA] *Appears in:* - [TLSCipherSuite](#tlsciphersuite) | Value | Description | | ----------------------------------------------- | ----------- | | `TLS_AES_256_GCM_SHA384` | TLS 1.3 | | `TLS_CHACHA20_POLY1305_SHA256` | | | `TLS_AES_128_GCM_SHA256` | | | `TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384` | TLS 1.2 | | `TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384` | | | `TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256` | | | `TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256` | | | `TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256` | | | `TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256` | | | `TLS_RSA_WITH_AES_256_GCM_SHA384` | | | `TLS_RSA_WITH_AES_128_GCM_SHA256` | | | `TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA` | | | `TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA` | | | `TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA` | | ### TLSCipherSuite[​](#tlsciphersuite) *Appears in:* - [TLSCipherSuites](#tlsciphersuites) | Field | Description | | -------------------------------- | -------------------------------------------------------- | | `name` *[TLSCipher](#tlscipher)* | (Optional) This should be a valid TLS cipher suite name. | ### TLSCipherSuites[​](#tlsciphersuites) *Underlying type:* *[TLSCipherSuite](#tlsciphersuite)* *Appears in:* - [InstallationSpec](#installationspec) | Field | Description | | -------------------------------- | -------------------------------------------------------- | | `name` *[TLSCipher](#tlscipher)* | (Optional) This should be a valid TLS cipher suite name. | ### TLSPassThroughRouteSpec[​](#tlspassthroughroutespec) *Appears in:* - [TLSPassThroughRoute](#tlspassthroughroute) | Field | Description | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `target` *[TargetType](#targettype)* | | | `sniMatch` *[SNIMatch](#snimatch)* | SNIMatch is used to match requests based on the server name for the intended destination server. Matching requests will be proxied to the Destination. | | `destination` *string* | Destination is the destination url to proxy the request to. | ### TLSTerminatedRouteSpec[​](#tlsterminatedroutespec) *Appears in:* - [TLSTerminatedRoute](#tlsterminatedroute) | Field | Description | | -------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `target` *[TargetType](#targettype)* | | | `pathMatch` *[PathMatch](#pathmatch)* | PathMatch is used to match requests based on what's in the path. Matching requests will be proxied to the Destination defined in this structure. | | `destination` *string* | Destination is the destination URL where matching traffic is routed to. | | `caBundle` *[ConfigMapKeySelector](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#configmapkeyselector-v1-core)* | CABundle is where we read the CA bundle from to authenticate the destination (if non-empty) | | `mtlsCert` *[SecretKeySelector](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#secretkeyselector-v1-core)* | (Optional) ForwardingMTLSCert is the certificate used for mTLS between voltron and the destination. Either both ForwardingMTLSCert and ForwardingMTLSKey must be specified, or neither can be specified. | | `mtlsKey` *[SecretKeySelector](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#secretkeyselector-v1-core)* | (Optional) ForwardingMTLSKey is the key used for mTLS between voltron and the destination. Either both ForwardingMTLSCert and ForwardingMTLSKey must be specified, or neither can be specified. | | `unauthenticated` *boolean* | (Optional) Unauthenticated says whether the request should go through authentication. This is only applicable if the Target is UI. | ### TargetType[​](#targettype) *Underlying type:* *string* *Appears in:* - [TLSPassThroughRouteSpec](#tlspassthroughroutespec) - [TLSTerminatedRouteSpec](#tlsterminatedroutespec) | Value | Description | | ---------------- | ----------- | | `UpstreamTunnel` | | | `UI` | | ### TigeraStatus[​](#tigerastatus) TigeraStatus represents the most recently observed status for Calico or a Calico Enterprise functional area. | Field | Description | | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------- | | `apiVersion` *string* | `operator.tigera.io/v1` | | `kind` *string* | `TigeraStatus` | | `metadata` *[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#objectmeta-v1-meta)* | Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[TigeraStatusSpec](#tigerastatusspec)* | | | `status` *[TigeraStatusStatus](#tigerastatusstatus)* | | ### TigeraStatusCondition[​](#tigerastatuscondition) TigeraStatusCondition represents a condition attached to a particular component. *Appears in:* - [TigeraStatusStatus](#tigerastatusstatus) | Field | Description | | ---------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `type` *[StatusConditionType](#statusconditiontype)* | The type of condition. May be Available, Progressing, or Degraded. | | `status` *[ConditionStatus](#conditionstatus)* | The status of the condition. May be True, False, or Unknown. | | `lastTransitionTime` *[Time](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#time-v1-meta)* | The timestamp representing the start time for the current status. | | `reason` *string* | A brief reason explaining the condition. | | `message` *string* | Optionally, a detailed message providing additional context. | | `observedGeneration` *integer* | (Optional) observedGeneration represents the generation that the condition was set based upon. For instance, if generation is currently 12, but the .status.conditions\[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance. | ### TigeraStatusSpec[​](#tigerastatusspec) TigeraStatusSpec defines the desired state of TigeraStatus *Appears in:* - [TigeraStatus](#tigerastatus) ### TigeraStatusStatus[​](#tigerastatusstatus) TigeraStatusStatus defines the observed state of TigeraStatus *Appears in:* - [TigeraStatus](#tigerastatus) | Field | Description | | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `conditions` *[TigeraStatusCondition](#tigerastatuscondition) array* | Conditions represents the latest observed set of conditions for this component. A component may be one or more of Available, Progressing, or Degraded. | ### TyphaAffinity[​](#typhaaffinity) Deprecated. Please use TyphaDeployment instead. TyphaAffinity allows configuration of node affinity characteristics for Typha pods. *Appears in:* - [InstallationSpec](#installationspec) | Field | Description | | ---------------------------------------------- | --------------------------------------------------------------------------- | | `nodeAffinity` *[NodeAffinity](#nodeaffinity)* | (Optional) NodeAffinity describes node affinity scheduling rules for typha. | ### TyphaDeployment[​](#typhadeployment) TyphaDeployment is the configuration for the typha Deployment. *Appears in:* - [InstallationSpec](#installationspec) | Field | Description | | ---------------------------------------------------- | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[TyphaDeploymentSpec](#typhadeploymentspec)* | (Optional) Spec is the specification of the typha Deployment. | ### TyphaDeploymentContainer[​](#typhadeploymentcontainer) TyphaDeploymentContainer is a typha Deployment container. *Appears in:* - [TyphaDeploymentPodSpec](#typhadeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `name` *string* | Name is an enum which identifies the typha Deployment container by name. Supported values are: calico-typha | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named typha Deployment container's resources. If omitted, the typha Deployment will use its default value for this container's resources. If used in conjunction with the deprecated ComponentResources, then this value takes precedence. | ### TyphaDeploymentInitContainer[​](#typhadeploymentinitcontainer) TyphaDeploymentInitContainer is a typha Deployment init container. *Appears in:* - [TyphaDeploymentPodSpec](#typhadeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` *string* | Name is an enum which identifies the typha Deployment init container by name. Supported values are: typha-certs-key-cert-provisioner | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named typha Deployment init container's resources. If omitted, the typha Deployment will use its default value for this init container's resources. If used in conjunction with the deprecated ComponentResources, then this value takes precedence. | ### TyphaDeploymentPodSpec[​](#typhadeploymentpodspec) TyphaDeploymentPodSpec is the typha Deployment's PodSpec. *Appears in:* - [TyphaDeploymentPodTemplateSpec](#typhadeploymentpodtemplatespec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `initContainers` *[TyphaDeploymentInitContainer](#typhadeploymentinitcontainer) array* | (Optional) InitContainers is a list of typha init containers. If specified, this overrides the specified typha Deployment init containers. If omitted, the typha Deployment will use its default values for its init containers. | | `containers` *[TyphaDeploymentContainer](#typhadeploymentcontainer) array* | (Optional) Containers is a list of typha containers. If specified, this overrides the specified typha Deployment containers. If omitted, the typha Deployment will use its default values for its containers. | | `affinity` *[Affinity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#affinity-v1-core)* | (Optional) Affinity is a group of affinity scheduling rules for the typha pods. If specified, this overrides any affinity that may be set on the typha Deployment. If omitted, the typha Deployment will use its default value for affinity. If used in conjunction with the deprecated TyphaAffinity, then this value takes precedence. WARNING: Please note that this field will override the default calico-typha Deployment affinity. | | `nodeSelector` *object (keys:string, values:string)* | NodeSelector is the calico-typha pod's scheduling constraints. If specified, each of the key/value pairs are added to the calico-typha Deployment nodeSelector provided the key does not already exist in the object's nodeSelector. If omitted, the calico-typha Deployment will use its default value for nodeSelector. WARNING: Please note that this field will modify the default calico-typha Deployment nodeSelector. | | `terminationGracePeriodSeconds` *integer* | (Optional) Optional duration in seconds the pod needs to terminate gracefully. May be decreased in delete request. Value must be non-negative integer. The value zero indicates stop immediately via the kill signal (no opportunity to shut down). If this value is nil, the default grace period will be used instead. The grace period is the duration in seconds after the processes running in the pod are sent a termination signal and the time when the processes are forcibly halted with a kill signal. Set this value longer than the expected cleanup time for your process. Defaults to 30 seconds. | | `topologySpreadConstraints` *[TopologySpreadConstraint](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#topologyspreadconstraint-v1-core) array* | (Optional) TopologySpreadConstraints describes how a group of pods ought to spread across topology domains. Scheduler will schedule pods in a way which abides by the constraints. All topologySpreadConstraints are ANDed. | | `tolerations` *[Toleration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#toleration-v1-core) array* | (Optional) Tolerations is the typha pod's tolerations. If specified, this overrides any tolerations that may be set on the typha Deployment. If omitted, the typha Deployment will use its default value for tolerations. WARNING: Please note that this field will override the default calico-typha Deployment tolerations. | ### TyphaDeploymentPodTemplateSpec[​](#typhadeploymentpodtemplatespec) TyphaDeploymentPodTemplateSpec is the typha Deployment's PodTemplateSpec *Appears in:* - [TyphaDeploymentSpec](#typhadeploymentspec) | Field | Description | | ---------------------------------------------------------- | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[TyphaDeploymentPodSpec](#typhadeploymentpodspec)* | (Optional) Spec is the typha Deployment's PodSpec. | ### TyphaDeploymentSpec[​](#typhadeploymentspec) TyphaDeploymentSpec defines configuration for the typha Deployment. *Appears in:* - [TyphaDeployment](#typhadeployment) | Field | Description | | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `minReadySeconds` *integer* | (Optional) MinReadySeconds is the minimum number of seconds for which a newly created Deployment pod should be ready without any of its container crashing, for it to be considered available. If specified, this overrides any minReadySeconds value that may be set on the typha Deployment. If omitted, the typha Deployment will use its default value for minReadySeconds. | | `template` *[TyphaDeploymentPodTemplateSpec](#typhadeploymentpodtemplatespec)* | (Optional) Template describes the typha Deployment pod that will be created. | | `strategy` *[TyphaDeploymentStrategy](#typhadeploymentstrategy)* | (Optional) The deployment strategy to use to replace existing pods with new ones. | ### TyphaDeploymentStrategy[​](#typhadeploymentstrategy) TyphaDeploymentStrategy describes how to replace existing pods with new ones. Only RollingUpdate is supported at this time so the Type field is not exposed. *Appears in:* - [TyphaDeploymentSpec](#typhadeploymentspec) | Field | Description | | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | `rollingUpdate` *[RollingUpdateDeployment](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#rollingupdatedeployment-v1-apps)* | (Optional) Rolling update config params. Present only if DeploymentStrategyType = RollingUpdate. to be. | ### UserMatch[​](#usermatch) UserMatch when the value of a UserAttribute and a GroupAttribute match, a user belongs to the group. *Appears in:* - [GroupSearch](#groupsearch) | Field | Description | | ------------------------- | ------------------------------------------------- | | `userAttribute` *string* | The attribute of a user that links it to a group. | | `groupAttribute` *string* | The attribute of a group that links it to a user. | ### UserSearch[​](#usersearch) User entry search configuration to match the credentials with a user. *Appears in:* - [AuthenticationLDAP](#authenticationldap) | Field | Description | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ | | `baseDN` *string* | BaseDN to start the search from. For example "cn=users,dc=example,dc=com" | | `filter` *string* | (Optional) Optional filter to apply when searching the directory. For example "(objectClass=person)" | | `nameAttribute` *string* | (Optional) A mapping of the attribute that is used as the username. This attribute can be used to apply RBAC to a user. Default: uid | ### WAFStatusType[​](#wafstatustype) *Underlying type:* *string* *Validation:* - Enum: \[Enabled Disabled] *Appears in:* - [ApplicationLayerSpec](#applicationlayerspec) | Value | Description | | ---------- | ----------- | | `Disabled` | | | `Enabled` | | ### Whisker[​](#whisker) | Field | Description | | ------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------- | | `apiVersion` *string* | `operator.tigera.io/v1` | | `kind` *string* | `Whisker` | | `metadata` *[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#objectmeta-v1-meta)* | Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[WhiskerSpec](#whiskerspec)* | | | `status` *[WhiskerStatus](#whiskerstatus)* | | ### WhiskerDeployment[​](#whiskerdeployment) WhiskerDeployment is the configuration for the whisker Deployment. *Appears in:* - [WhiskerSpec](#whiskerspec) | Field | Description | | -------------------------------------------------------- | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[WhiskerDeploymentSpec](#whiskerdeploymentspec)* | (Optional) Spec is the specification of the whisker Deployment. | ### WhiskerDeploymentContainer[​](#whiskerdeploymentcontainer) *Appears in:* - [WhiskerDeploymentPodSpec](#whiskerdeploymentpodspec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------- | ----------- | | `name` *string* | | | `resources` *[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#resourcerequirements-v1-core)* | (Optional) | ### WhiskerDeploymentPodSpec[​](#whiskerdeploymentpodspec) WhiskerDeploymentPodSpec is the whisker Deployment's PodSpec. *Appears in:* - [WhiskerDeploymentPodTemplateSpec](#whiskerdeploymentpodtemplatespec) | Field | Description | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `affinity` *[Affinity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#affinity-v1-core)* | (Optional) Affinity is a group of affinity scheduling rules for the whisker pods. | | `containers` *[WhiskerDeploymentContainer](#whiskerdeploymentcontainer) array* | (Optional) Containers is a list of whisker containers. If specified, this overrides the specified EGW Deployment containers. If omitted, the whisker Deployment will use its default values for its containers. | | `nodeSelector` *object (keys:string, values:string)* | (Optional) NodeSelector gives more control over the nodes where the whisker pods will run on. | | `terminationGracePeriodSeconds` *integer* | (Optional) TerminationGracePeriodSeconds defines the termination grace period of the whisker pods in seconds. | | `topologySpreadConstraints` *[TopologySpreadConstraint](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#topologyspreadconstraint-v1-core) array* | (Optional) TopologySpreadConstraints describes how a group of pods ought to spread across topology domains. Scheduler will schedule pods in a way which abides by the constraints. All topologySpreadConstraints are ANDed. | | `tolerations` *[Toleration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#toleration-v1-core) array* | (Optional) Tolerations is the whisker pod's tolerations. If specified, this overrides any tolerations that may be set on the whisker Deployment. If omitted, the whisker Deployment will use its default value for tolerations. | | `priorityClassName` *string* | (Optional) PriorityClassName allows to specify a PriorityClass resource to be used. | ### WhiskerDeploymentPodTemplateSpec[​](#whiskerdeploymentpodtemplatespec) WhiskerDeploymentPodTemplateSpec is the whisker Deployment's PodTemplateSpec *Appears in:* - [WhiskerDeploymentSpec](#whiskerdeploymentspec) | Field | Description | | -------------------------------------------------------------- | -------------------------------------------------------------------------- | | `metadata` *[Metadata](#metadata)* | (Optional) Refer to Kubernetes API documentation for fields of `metadata`. | | `spec` *[WhiskerDeploymentPodSpec](#whiskerdeploymentpodspec)* | (Optional) Spec is the whisker Deployment's PodSpec. | ### WhiskerDeploymentSpec[​](#whiskerdeploymentspec) WhiskerDeploymentSpec defines configuration for the whisker Deployment. *Appears in:* - [WhiskerDeployment](#whiskerdeployment) | Field | Description | | ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `minReadySeconds` *integer* | (Optional) MinReadySeconds is the minimum number of seconds for which a newly created Deployment pod should be ready without any of its container crashing, for it to be considered available. If specified, this overrides any minReadySeconds value that may be set on the whisker Deployment. If omitted, the whisker Deployment will use its default value for minReadySeconds. | | `template` *[WhiskerDeploymentPodTemplateSpec](#whiskerdeploymentpodtemplatespec)* | (Optional) Template describes the whisker Deployment pod that will be created. | | `strategy` *[WhiskerDeploymentStrategy](#whiskerdeploymentstrategy)* | (Optional) The deployment strategy to use to replace existing pods with new ones. | ### WhiskerDeploymentStrategy[​](#whiskerdeploymentstrategy) *Appears in:* - [WhiskerDeploymentSpec](#whiskerdeploymentspec) | Field | Description | | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | `rollingUpdate` *[RollingUpdateDeployment](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#rollingupdatedeployment-v1-apps)* | (Optional) Rolling update config params. Present only if DeploymentStrategyType = RollingUpdate. to be. | ### WhiskerSpec[​](#whiskerspec) *Appears in:* - [Whisker](#whisker) | Field | Description | | ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `whiskerDeployment` *[WhiskerDeployment](#whiskerdeployment)* | | | `notifications` *[NotificationMode](#notificationmode)* | (Optional) Default: Enabled This setting enables calls to an external API to retrieve notification banner text in the Whisker UI. Allowed values are Enabled or Disabled. Defaults to Enabled. | ### WhiskerStatus[​](#whiskerstatus) WhiskerStatus defines the observed state of Whisker *Appears in:* - [Whisker](#whisker) | Field | Description | | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `conditions` *[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.32/#condition-v1-meta) array* | (Optional) Conditions represents the latest observed set of conditions for the component. A component may be one or more of Ready, Progressing, Degraded or other customer types. | ### WindowsDataplaneOption[​](#windowsdataplaneoption) *Underlying type:* *string* *Validation:* - Enum: \[HNS Disabled] *Appears in:* - [CalicoNetworkSpec](#caliconetworkspec) | Value | Description | | ---------- | ----------- | | `Disabled` | | | `HNS` | | ### WindowsNodeSpec[​](#windowsnodespec) *Appears in:* - [InstallationSpec](#installationspec) | Field | Description | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `cniBinDir` *string* | (Optional) CNIBinDir is the path to the CNI binaries directory on Windows, it must match what is used as 'bin\_dir' under \[plugins] \[plugins."io.containerd.grpc.v1.cri"] \[plugins."io.containerd.grpc.v1.cri".cni] on the containerd 'config.toml' file on the Windows nodes. | | `cniConfigDir` *string* | (Optional) CNIConfigDir is the path to the CNI configuration directory on Windows, it must match what is used as 'conf\_dir' under \[plugins] \[plugins."io.containerd.grpc.v1.cri"] \[plugins."io.containerd.grpc.v1.cri".cni] on the containerd 'config.toml' file on the Windows nodes. | | `cniLogDir` *string* | (Optional) CNILogDir is the path to the Calico CNI logs directory on Windows. | | `vxlanMACPrefix` *string* | (Optional) VXLANMACPrefix is the prefix used when generating MAC addresses for virtual NICs | | `vxlanAdapter` *string* | (Optional) VXLANAdapter is the Network Adapter used for VXLAN, leave blank for primary NIC | ### Image Assurance Installation reference ## Image Assurance installation reference[​](#image-assurance-installation-reference-1) The Kubernetes resources below configure Calico Cloud Image Assurance installation when using the operator. Each resource is responsible for installing and configuring a different subsystem of Calico Cloud Image Assurance during installation. Most options can be modified on a running cluster using `kubectl`. Packages: - [image-assurance.operator.tigera.io/v1](#image-assurance.operator.tigera.io%2fv1) ## image-assurance.operator.tigera.io/v1[#image-assurance.operator.tigera.io%2fv1](#image-assurance.operator.tigera.io%2fv1) API Schema definitions for configuring the installation of Image Assurance Resource Types: ### ClusterScannerStatusType(`string` alias)[#image-assurance.operator.tigera.io%2fv1.ClusterScannerStatusType](#image-assurance.operator.tigera.io%2fv1.ClusterScannerStatusType) (*Appears on:* [ImageAssuranceSpec](#image-assurance.operator.tigera.io/v1.ImageAssuranceSpec)) ### CrawdadDaemonSet[#image-assurance.operator.tigera.io%2fv1.CrawdadDaemonSet](#image-assurance.operator.tigera.io%2fv1.CrawdadDaemonSet) (*Appears on:* [ImageAssuranceSpec](#image-assurance.operator.tigera.io/v1.ImageAssuranceSpec)) | Field | Description | | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | `metadata` ***github.com/tigera/operator/api/v1.Metadata* | *(Optional)*Metadata is a subset of a Kubernetes object’s metadata that is added to the DaemonSet. | | `spec` *[CrawdadDaemonSetSpec](#image-assurance.operator.tigera.io/v1.CrawdadDaemonSetSpec)* | *(Optional)*Spec is the specification of the crawdad DaemonSet. | ### CrawdadDaemonSetContainer[#image-assurance.operator.tigera.io%2fv1.CrawdadDaemonSetContainer](#image-assurance.operator.tigera.io%2fv1.CrawdadDaemonSetContainer) (*Appears on:* [CrawdadDaemonSetPodSpec](#image-assurance.operator.tigera.io/v1.CrawdadDaemonSetPodSpec)) CrawdadDaemonSetContainer is a crawdad DaemonSet container. | Field | Description | | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` ***string* | Name is an enum which identifies the crawdad DaemonSet container by name. | | `resources` *[Kubernetes core/v1.ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#resourcerequirements-v1-core)* | *(Optional)*Resources allows customization of limits and requests for compute resources such as cpu and memory. If specified, this overrides the named crawdad DaemonSet container’s resources. If omitted, the crawdad DaemonSet will use its default value for this container’s resources. If used in conjunction with the deprecated ComponentResources, then this value takes precedence. | ### CrawdadDaemonSetPodSpec[#image-assurance.operator.tigera.io%2fv1.CrawdadDaemonSetPodSpec](#image-assurance.operator.tigera.io%2fv1.CrawdadDaemonSetPodSpec) (*Appears on:* [CrawdadDaemonSetPodTemplateSpec](#image-assurance.operator.tigera.io/v1.CrawdadDaemonSetPodTemplateSpec)) CrawdadDaemonSetPodSpec is the crawdad DaemonSet’s PodSpec. | Field | Description | | -------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `containers` *[\[\]CrawdadDaemonSetContainer](#image-assurance.operator.tigera.io/v1.CrawdadDaemonSetContainer)* | *(Optional)*Containers is a list of crawdad containers. If specified, this overrides the specified crawdad DaemonSet cluster-scanner containers. If omitted, the crawdad DaemonSet will use its default values for its containers. | | `affinity` *[Kubernetes core/v1.Affinity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#affinity-v1-core)* | *(Optional)*Affinity is a group of affinity scheduling rules for the crawdad pods. If specified, this overrides any affinity that may be set on the crawdad DaemonSet. If omitted, the crawdad DaemonSet will use its default value for affinity. WARNING: Please note that this field will override the default crawdad DaemonSet affinity. | | `nodeSelector` ***map\[string]string* | NodeSelector is the crawdad pod’s scheduling constraints. If specified, each of the key/value pairs are added to the crawdad DaemonSet nodeSelector provided the key does not already exist in the object’s nodeSelector. If used in conjunction with ControlPlaneNodeSelector, that nodeSelector is set on the crawdad DaemonSet and each of this field’s key/value pairs are added to the crawdad DaemonSet nodeSelector provided the key does not already exist in the object’s nodeSelector. If omitted, the crawdad DaemonSet will use its default value for nodeSelector. WARNING: Please note that this field will modify the default crawdad DaemonSet nodeSelector. | | `tolerations` *[\[\]Kubernetes core/v1.Toleration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#toleration-v1-core)* | *(Optional)*Tolerations is the crawdad pod’s tolerations. If specified, this overrides any tolerations that may be set on the crawdad DaemonSet. If omitted, the crawdad DaemonSet will use its default value for tolerations. WARNING: Please note that this field will override the default crawdad DaemonSet tolerations. | ### CrawdadDaemonSetPodTemplateSpec[#image-assurance.operator.tigera.io%2fv1.CrawdadDaemonSetPodTemplateSpec](#image-assurance.operator.tigera.io%2fv1.CrawdadDaemonSetPodTemplateSpec) (*Appears on:* [CrawdadDaemonSetSpec](#image-assurance.operator.tigera.io/v1.CrawdadDaemonSetSpec)) CrawdadDaemonSetPodTemplateSpec is the crawdad DaemonSet’s PodTemplateSpec | Field | Description | | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | | `metadata` ***github.com/tigera/operator/api/v1.Metadata* | *(Optional)*Metadata is a subset of a Kubernetes object’s metadata that is added to the pod’s metadata. | | `spec` *[CrawdadDaemonSetPodSpec](#image-assurance.operator.tigera.io/v1.CrawdadDaemonSetPodSpec)* | *(Optional)*Spec is the crawdad DaemonSet’s PodSpec. | ### CrawdadDaemonSetSpec[#image-assurance.operator.tigera.io%2fv1.CrawdadDaemonSetSpec](#image-assurance.operator.tigera.io%2fv1.CrawdadDaemonSetSpec) (*Appears on:* [CrawdadDaemonSet](#image-assurance.operator.tigera.io/v1.CrawdadDaemonSet)) CrawdadDaemonSetSpec defines configuration for the crawdad DaemonSet. | Field | Description | | ---------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `minReadySeconds` ***int32* | *(Optional)*MinReadySeconds is the minimum number of seconds for which a newly created DaemonSet pod should be ready without any of its container crashing, for it to be considered available. If specified, this overrides any minReadySeconds value that may be set on the crawdad DaemonSet. If omitted, the crawdad DaemonSet will use its default value for minReadySeconds. | | `template` *[CrawdadDaemonSetPodTemplateSpec](#image-assurance.operator.tigera.io/v1.CrawdadDaemonSetPodTemplateSpec)* | *(Optional)*Template describes the crawdad DaemonSet pod that will be created. | ### ExcludedNamespace(`string` alias)[#image-assurance.operator.tigera.io%2fv1.ExcludedNamespace](#image-assurance.operator.tigera.io%2fv1.ExcludedNamespace) (*Appears on:* [Exclusions](#image-assurance.operator.tigera.io/v1.Exclusions)) ExcludedNamespace is a namespace name to be excluded from image scanning. ### Exclusions[#image-assurance.operator.tigera.io%2fv1.Exclusions](#image-assurance.operator.tigera.io%2fv1.Exclusions) (*Appears on:* [ImageAssuranceSpec](#image-assurance.operator.tigera.io/v1.ImageAssuranceSpec)) Exclusions specifies the criteria for what to exclude from image scanning. | Field | Description | | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------- | | `namespaces` *[\[\]ExcludedNamespace](#image-assurance.operator.tigera.io/v1.ExcludedNamespace)* | *(Optional)*Namespaces is an array of namespace names to be excluded from image scanning. | ### ImageAssurance[#image-assurance.operator.tigera.io%2fv1.ImageAssurance](#image-assurance.operator.tigera.io%2fv1.ImageAssurance) ImageAssurance is the Schema for the imageassurances API | Field | Description | | ------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `metadata` *[Kubernetes meta/v1.ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#objectmeta-v1-meta)* | Refer to the Kubernetes API documentation for the fields of the `metadata` field. | | `spec` *[ImageAssuranceSpec](#image-assurance.operator.tigera.io/v1.ImageAssuranceSpec)* | criSocketPath string CRISocketPath is the path to the CRI socket on the nodes. Defaults to /run/containerd/containerd.sock. containerdVolumeMountPath string (Optional) ContainerdVolumeMountPath is the path to the root of containerd file system. Defaults to /var/lib/containerd/. clusterScanner ClusterScannerStatusType (Optional) This setting enables or disables the cluster scanner. Allowed values are Enabled or Disabled. Defaults to Disabled. crawdadDaemonset CrawdadDaemonSet (Optional) CrawdadDaemonSet is the specification of the Crawdad Daemonset. exclusions Exclusions (Optional) Exclusions define the exclusion criteria for image scanning. Note: Exclusions are applied to future scans and do not affect past scan results. | | `status` *[ImageAssuranceStatus](#image-assurance.operator.tigera.io/v1.ImageAssuranceStatus)* | | ### ImageAssuranceSpec[#image-assurance.operator.tigera.io%2fv1.ImageAssuranceSpec](#image-assurance.operator.tigera.io%2fv1.ImageAssuranceSpec) (*Appears on:* [ImageAssurance](#image-assurance.operator.tigera.io/v1.ImageAssurance)) ImageAssuranceSpec configures Image Assurance monitoring and tooling in a kubernetes cluster. | Field | Description | | -------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `criSocketPath` ***string* | CRISocketPath is the path to the CRI socket on the nodes. Defaults to /run/containerd/containerd.sock. | | `containerdVolumeMountPath` ***string* | *(Optional)*ContainerdVolumeMountPath is the path to the root of containerd file system. Defaults to /var/lib/containerd/. | | `clusterScanner` *[ClusterScannerStatusType](#image-assurance.operator.tigera.io/v1.ClusterScannerStatusType)* | *(Optional)*This setting enables or disables the cluster scanner. Allowed values are Enabled or Disabled. Defaults to Disabled. | | `crawdadDaemonset` *[CrawdadDaemonSet](#image-assurance.operator.tigera.io/v1.CrawdadDaemonSet)* | *(Optional)*CrawdadDaemonSet is the specification of the Crawdad Daemonset. | | `exclusions` *[Exclusions](#image-assurance.operator.tigera.io/v1.Exclusions)* | *(Optional)*Exclusions define the exclusion criteria for image scanning. Note: Exclusions are applied to future scans and do not affect past scan results. | ### ImageAssuranceStatus[#image-assurance.operator.tigera.io%2fv1.ImageAssuranceStatus](#image-assurance.operator.tigera.io%2fv1.ImageAssuranceStatus) (*Appears on:* [ImageAssurance](#image-assurance.operator.tigera.io/v1.ImageAssurance)) ImageAssuranceStatus defines the observed state of ImageAssurance *** ### Resource definitions ## [📄️ Resource definitions](https://docs.tigera.io/calico-cloud/reference/resources/overview) [Calico Cloud resources (APIs) that you can manage using calicoctl.](https://docs.tigera.io/calico-cloud/reference/resources/overview) ## [📄️ BFD configuration](https://docs.tigera.io/calico-cloud/reference/resources/bfdconfig) [API for this Calico Enterprise resource.](https://docs.tigera.io/calico-cloud/reference/resources/bfdconfig) ## [📄️ BGP configuration](https://docs.tigera.io/calico-cloud/reference/resources/bgpconfig) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/bgpconfig) ## [📄️ BGP peer](https://docs.tigera.io/calico-cloud/reference/resources/bgppeer) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/bgppeer) ## [📄️ BGP Filter](https://docs.tigera.io/calico-cloud/reference/resources/bgpfilter) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/bgpfilter) ## [📄️ Block affinity](https://docs.tigera.io/calico-cloud/reference/resources/blockaffinity) [IP address management block affinity](https://docs.tigera.io/calico-cloud/reference/resources/blockaffinity) ## [📄️ Calico node status](https://docs.tigera.io/calico-cloud/reference/resources/caliconodestatus) [API for this Calico resource.](https://docs.tigera.io/calico-cloud/reference/resources/caliconodestatus) ## [📄️ Container admission policy](https://docs.tigera.io/calico-cloud/reference/resources/containeradmissionpolicy) [Resource definition.](https://docs.tigera.io/calico-cloud/reference/resources/containeradmissionpolicy) ## [🗃️ Compliance reports](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/) [5 items](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/) ## [📄️ Deep packet inspection](https://docs.tigera.io/calico-cloud/reference/resources/deeppacketinspection) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/deeppacketinspection) ## [📄️ Felix configuration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig) [API for this Calico Enterprise resource.](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig) ## [📄️ Egress gateway policy](https://docs.tigera.io/calico-cloud/reference/resources/egressgatewaypolicy) [API for this Calico Enterprise resource.](https://docs.tigera.io/calico-cloud/reference/resources/egressgatewaypolicy) ## [📄️ Global Alert](https://docs.tigera.io/calico-cloud/reference/resources/globalalert) [API for this Calico Enterprise resource.](https://docs.tigera.io/calico-cloud/reference/resources/globalalert) ## [📄️ Global network policy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) ## [📄️ Global network set](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkset) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkset) ## [📄️ Global report](https://docs.tigera.io/calico-cloud/reference/resources/globalreport) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/globalreport) ## [📄️ Global threat feed](https://docs.tigera.io/calico-cloud/reference/resources/globalthreatfeed) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/globalthreatfeed) ## [📄️ Host endpoint](https://docs.tigera.io/calico-cloud/reference/resources/hostendpoint) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/hostendpoint) ## [📄️ IP pool](https://docs.tigera.io/calico-cloud/reference/resources/ippool) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/ippool) ## [📄️ IP reservation](https://docs.tigera.io/calico-cloud/reference/resources/ipreservation) [API for this Calico resource.](https://docs.tigera.io/calico-cloud/reference/resources/ipreservation) ## [📄️ IPAM configuration](https://docs.tigera.io/calico-cloud/reference/resources/ipamconfig) [IP address management global configuration](https://docs.tigera.io/calico-cloud/reference/resources/ipamconfig) ## [📄️ License key](https://docs.tigera.io/calico-cloud/reference/resources/licensekey) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/licensekey) ## [📄️ Kubernetes controllers configuration](https://docs.tigera.io/calico-cloud/reference/resources/kubecontrollersconfig) [API for KubeControllersConfiguration resource.](https://docs.tigera.io/calico-cloud/reference/resources/kubecontrollersconfig) ## [📄️ Managed Cluster](https://docs.tigera.io/calico-cloud/reference/resources/managedcluster) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/managedcluster) ## [📄️ Network policy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) ## [📄️ Network set](https://docs.tigera.io/calico-cloud/reference/resources/networkset) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/networkset) ## [📄️ Node](https://docs.tigera.io/calico-cloud/reference/resources/node) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/node) ## [📄️ PacketCapture](https://docs.tigera.io/calico-cloud/reference/resources/packetcapture) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/packetcapture) ## [📄️ Remote cluster configuration](https://docs.tigera.io/calico-cloud/reference/resources/remoteclusterconfiguration) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/remoteclusterconfiguration) ## [📄️ RuntimeSecurity](https://docs.tigera.io/calico-cloud/reference/resources/runtimesecurity) [API for this Calico Cloud resource](https://docs.tigera.io/calico-cloud/reference/resources/runtimesecurity) ## [📄️ Security event webhook](https://docs.tigera.io/calico-cloud/reference/resources/securityeventwebhook) [API for this Calico Enterprise resource.](https://docs.tigera.io/calico-cloud/reference/resources/securityeventwebhook) ## [📄️ Staged global network policy](https://docs.tigera.io/calico-cloud/reference/resources/stagedglobalnetworkpolicy) [API for this resource.](https://docs.tigera.io/calico-cloud/reference/resources/stagedglobalnetworkpolicy) ## [📄️ Staged Kubernetes network policy](https://docs.tigera.io/calico-cloud/reference/resources/stagedkubernetesnetworkpolicy) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/stagedkubernetesnetworkpolicy) ## [📄️ Staged network policy](https://docs.tigera.io/calico-cloud/reference/resources/stagednetworkpolicy) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/stagednetworkpolicy) ## [📄️ Tier](https://docs.tigera.io/calico-cloud/reference/resources/tier) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/tier) ## [📄️ Workload endpoint](https://docs.tigera.io/calico-cloud/reference/resources/workloadendpoint) [API for this Calico Cloud resource.](https://docs.tigera.io/calico-cloud/reference/resources/workloadendpoint) ### Resource definitions This section describes the set of valid resource types that can be managed through `calicoctl` or `kubectl`. While resources may be supplied in YAML or JSON format, this guide provides examples in YAML. ## Overview of resource structure[​](#overview-of-resource-structure) The calicoctl commands for resource management (create, apply, delete, replace, get) all take resource manifests as input. Each manifest may contain a single resource (e.g. a profile resource), or a list of multiple resources (e.g. a profile and two hostEndpoint resources). The general structure of a single resource is as follows: ```yaml apiVersion: projectcalico.org/v3 kind: metadata: # Identifying information name: ... spec: # Specification of the resource ... ``` ### Schema[​](#schema) | Field | Description | Accepted Values | Schema | | ---------- | --------------------------------------------------------------------------------------- | -------------------- | ------------------------ | | apiVersion | Indicates the version of the API that the data corresponds to. | projectcalico.org/v3 | string | | kind | Specifies the type of resource described by the YAML document. | | [kind](#supported-kinds) | | metadata | Contains information used to uniquely identify the particular instance of the resource. | | map | | spec | Contains the resource specification. | | map | ### Supported kinds[​](#supported-kinds) The following resources are supported: - [AlertException](https://docs.tigera.io/calico-cloud/reference/resources/alertexception) - [BGPConfiguration](https://docs.tigera.io/calico-cloud/reference/resources/bgpconfig) - [BGPPeer](https://docs.tigera.io/calico-cloud/reference/resources/bgppeer) - [DeepPacketInspection](https://docs.tigera.io/calico-cloud/reference/resources/deeppacketinspection) - [EgressGatewayPolicy](https://docs.tigera.io/calico-cloud/reference/resources/egressgatewaypolicy) - [FelixConfiguration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig) - [GlobalAlert](https://docs.tigera.io/calico-cloud/reference/resources/globalalert) - [GlobalNetworkPolicy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) - [GlobalNetworkSet](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkset) - [GlobalReport](https://docs.tigera.io/calico-cloud/reference/resources/globalreport) - [GlobalThreatFeed](https://docs.tigera.io/calico-cloud/reference/resources/globalthreatfeed) - [HostEndpoint](https://docs.tigera.io/calico-cloud/reference/resources/hostendpoint) - [IPPool](https://docs.tigera.io/calico-cloud/reference/resources/ippool) - [IPReservation](https://docs.tigera.io/calico-cloud/reference/resources/ipreservation) - [KubeControllersConfiguration](https://docs.tigera.io/calico-cloud/reference/resources/kubecontrollersconfig) - [LicenseKey](https://docs.tigera.io/calico-cloud/reference/resources/licensekey) - [ManagedCluster](https://docs.tigera.io/calico-cloud/reference/resources/managedcluster) - [NetworkPolicy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) - [NetworkSet](https://docs.tigera.io/calico-cloud/reference/resources/networkset) - [Node](https://docs.tigera.io/calico-cloud/reference/resources/node) - [PacketCapture](https://docs.tigera.io/calico-cloud/reference/resources/packetcapture) - [RemoteClusterConfiguration](https://docs.tigera.io/calico-cloud/reference/resources/remoteclusterconfiguration) - [StagedGlobalNetworkPolicy](https://docs.tigera.io/calico-cloud/reference/resources/stagedglobalnetworkpolicy) - [StagedKubernetesNetworkPolicy](https://docs.tigera.io/calico-cloud/reference/resources/stagedkubernetesnetworkpolicy) - [StagedNetworkPolicy](https://docs.tigera.io/calico-cloud/reference/resources/stagednetworkpolicy) - [Tier](https://docs.tigera.io/calico-cloud/reference/resources/tier) - [WorkloadEndpoint](https://docs.tigera.io/calico-cloud/reference/resources/workloadendpoint) ### Resource name requirements[​](#resource-name-requirements) Every resource must have the `name` field specified. Name must be unique within a namespace. Name required when creating resources, and cannot be updated. A valid resource name can have alphanumeric characters with optional `.`, `_`, or `-`. of up to 128 characters total. ### Multiple resources in a single file[​](#multiple-resources-in-a-single-file) A file may contain multiple resource documents specified in a YAML list format. For example, the following is the contents of a file containing two `HostEndpoint` resources: ```yaml - apiVersion: projectcalico.org/v3 kind: HostEndpoint metadata: name: endpoint1 labels: type: database spec: interface: eth0 node: host1 profiles: - prof1 - prof2 expectedIPs: - 1.2.3.4 - '00:bb::aa' - apiVersion: projectcalico.org/v3 kind: HostEndpoint metadata: name: endpoint2 labels: type: frontend spec: interface: eth1 node: host1 profiles: - prof1 - prof2 expectedIPs: - 1.2.3.5 ``` ### BFD configuration A bidirectional forwarding detection (BFD) configuration resource (`BFDConfiguration`) represents BFD-specific configuration options for a selection of one or more nodes. For `kubectl` commands, the following case-insensitive aliases may be used to specify the resource type on the CLI: `bfdconfiguration.projectcalico.org`, `bfdconfigurations.projectcalico.org` as well as abbreviations such as `bfdconfiguration.p` and `bfdconfigurations.p`. ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: BFDConfiguration metadata: name: default spec: nodeSelector: "all()" interfaces: - matchPattern: "*" minimumRecvInterval: 10ms minimumSendInterval: 100ms idleSendInterval: 1m multiplier: 5 ``` ## BFD configuration definition[​](#bfd-configuration-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | | ----- | --------------------------------------------------------- | --------------------------------------------------- | ------ | | name | Unique name to describe this resource instance. Required. | Alphanumeric string with optional `.`, `_`, or `-`. | string | ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | ------------ | --------------------------------------------------------- | --------------- | ------------------------------------- | --------------------- | | nodeSelector | Selects one or more nodes to which this resource applies. | | string | [selector](#selector) | | interfaces | List of per-interface BFD configuration parameters. | | List of [BFDInterface](#bfdinterface) | | ### BFDInterface[​](#bfdinterface) | Field | Description | Accepted Values | Schema | Default | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | ------ | ------- | | matchPaterrn | Pattern to match one or more interfaces. Supports exact interface names, match on interface prefixes (e.g., “eth\*”), or “\*” to select all interfaces on the selected node(s). | | string | | | minimumRecvInterval | The minimum interval between received BFD packets. Must be a whole number of milliseconds greater than 0. | | string | 10ms | | minimumSendInterval | The minimum interval between transmitted BFD packets. Must be a whole number of milliseconds greater than 0. | | string | 100ms | | idleSendInterval | The interval between transmitted BFD packets when the BFD peer is idle. Must be a whole number of milliseconds greater than 0. | | string | 1m | | multiplier | The number of intervals that must pass without receiving a BFD packet before the peer is considered down. | | string | 5 | ### Selector[​](#selector) A label selector is an expression which either matches or does not match a resource based on its labels. $\[prodname] label selectors support a number of operators, which can be combined into larger expressions using the boolean operators and parentheses. | Expression | Meaning | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Logical operators** | | | `( )` | Matches if and only if `` matches. (Parentheses are used for grouping expressions.) | | `! ` | Matches if and only if `` does not match. **Tip:** `!` is a special character at the start of a YAML string, if you need to use `!` at the start of a YAML string, enclose the string in quotes. | | ` && ` | "And": matches if and only if both ``, and, `` matches | | ` \|\| ` | "Or": matches if and only if either ``, or, `` matches. | | **Match operators** | | | `all()` | Match all in-scope resources. To match *no* resources, combine this operator with `!` to form `!all()`. | | `global()` | Match all non-namespaced resources. Useful in a `namespaceSelector` to select global resources such as global network sets. | | `k == 'v'` | Matches resources with the label 'k' and value 'v'. | | `k != 'v'` | Matches resources without label 'k' or with label 'k' and value *not* equal to `v` | | `has(k)` | Matches resources with label 'k', independent of value. To match pods that do not have label `k`, combine this operator with `!` to form `!has(k)` | | `k in { 'v1', 'v2' }` | Matches resources with label 'k' and value in the given set | | `k not in { 'v1', 'v2' }` | Matches resources without label 'k' or with label 'k' and value *not* in the given set | | `k contains 's'` | Matches resources with label 'k' and value containing the substring 's' | | `k starts with 's'` | Matches resources with label 'k' and value starting with the substring 's' | | `k ends with 's'` | Matches resources with label 'k' and value ending with the substring 's' | Operators have the following precedence: - **Highest**: all the match operators - Parentheses `( ... )` - Negation with `!` - Conjunction with `&&` - **Lowest**: Disjunction with `||` For example, the expression ```text ! has(my-label) || my-label starts with 'prod' && role in {'frontend','business'} ``` Would be "bracketed" like this: ```text ((!(has(my-label)) || ((my-label starts with 'prod') && (role in {'frontend','business'})) ``` It would match: - Any resource that did not have label "my-label". - Any resource that both: - Has a value for `my-label` that starts with "prod", and, - Has a role label with value either "frontend", or "business". ## Supported operations[​](#supported-operations) | Datastore type | Create | Delete | Update | Get/List | Notes | | --------------------- | ------ | ------ | ------ | -------- | ----- | | Kubernetes API server | Yes | Yes | Yes | Yes | | ### BGP configuration A BGP configuration resource (`BGPConfiguration`) represents BGP specific configuration options for the cluster or a specific node. For `kubectl` commands, the following case-insensitive aliases may be used to specify the resource type on the CLI: `bgpconfiguration.projectcalico.org`, `bgpconfigurations.projectcalico.org` as well as abbreviations such as `bgpconfiguration.p` and `bgpconfigurations.p`. ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: BGPConfiguration metadata: name: default spec: logSeverityScreen: Info nodeToNodeMeshEnabled: true nodeMeshMaxRestartTime: 120s asNumber: 63400 serviceClusterIPs: - cidr: 10.96.0.0/12 serviceExternalIPs: - cidr: 104.244.42.129/32 - cidr: 172.217.3.0/24 listenPort: 178 bindMode: NodeIP communities: - name: bgp-large-community value: 63400:300:100 prefixAdvertisements: - cidr: 172.218.4.0/26 communities: - bgp-large-community - 63400:120 ``` ## BGP configuration definition[​](#bgp-configuration-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | | ----- | --------------------------------------------------------- | --------------------------------------------------- | ------ | | name | Unique name to describe this resource instance. Required. | Alphanumeric string with optional `.`, `_`, or `-`. | string | - The resource with the name `default` has a specific meaning - this contains the BGP global default configuration. - The resources with the name `node.` contain the node-specific overrides, and will be applied to the node ``. When deleting a node the BGPConfiguration resource associated with the node will also be deleted. Only prefixAdvertisements, listenPort, and logSeverityScreen can be overridden this way. ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------- | ------------------------------------------------------------------------------------------ | --------------------------------------------------------------- | | logSeverityScreen | Global log level | Debug, Info, Warning, Error, Fatal | string | `Info` | | nodeToNodeMeshEnabled | Full BGP node-to-node mesh. Only valid on the global `default` BGPConfiguration. | true, false | string | true | | asNumber | The default local AS Number that Calico Cloud should use when speaking with BGP peers. Only valid on the global `default` BGPConfiguration; to set a per-node override, use the `bgp` field on the [Node resource](https://docs.tigera.io/calico-cloud/reference/resources/node). | A valid AS Number, may be specified in dotted notation. | integer/string | 64512 | | extensions | Additional mapping of keys and values. Used for setting values in custom BGP configurations. | valid strings for both keys and values | map | | | serviceClusterIPs | The CIDR blocks for Kubernetes Service Cluster IPs to be advertised over BGP. Only valid on the global `default` BGPConfiguration: will be ignored otherwise. | A list of valid IPv4 or IPv6 CIDR blocks. | List of `cidr: /` values. | Empty List | | serviceExternalIPs | The CIDR blocks for Kubernetes Service External IPs to be advertised over BGP. Kubernetes Service External IPs will only be advertised if they are within one of these blocks. Only valid on the global `default` BGPConfiguration: will be ignored otherwise. | A list of valid IPv4 or IPv6 CIDR blocks. | List of `cidr: /` values. | Empty List | | serviceLoadBalancerIPs | The CIDR blocks for Kubernetes Service status.LoadBalancer IPs to be advertised over BGP. Kubernetes LoadBalancer IPs will only be advertised if they are within one of these blocks. Only valid on the global `default` BGPConfiguration: will be ignored otherwise. | A list of valid IPv4 or IPv6 CIDR blocks. | List of `cidr: /` values. | Empty List | | listenPort | The port where BGP protocol should listen. | A valid port number. | integer | 179 | | bindMode | Indicates whether to listen for BGP connections on all addresses (None) or only on the node's canonical IP address Node.Spec.BGP.IPvXAddress (NodeIP). If this field is changed when calico-node is already running, the change will not take effect until calico-node is manually restarted. | None, NodeIP. | string | None | | communities | List of BGP community names and their values, communities are not advertised unless they are used in [prefixAdvertisements](#prefixadvertisements). | | List of [communities](#communities) | | | prefixAdvertisements | List of per-prefix advertisement properties, like BGP communities. | | List of [prefixAdvertisements](#prefixadvertisements) | | | nodeMeshPassword | BGP password for the all the peerings in a full mesh configuration. | | [BGPPassword](https://docs.tigera.io/calico-cloud/reference/resources/bgppeer#bgppassword) | `nil` (no password) | | nodeMeshMaxRestartTime | Restart time that is announced by BIRD in the BGP graceful restart capability and that specifies how long the neighbor would wait for the BGP session to re-establish after a restart before deleting stale routes in full mesh configurations. Note: extra care should be taken when changing this configuration, as it may break networking in your cluster. When not specified, BIRD uses the default value of 120 seconds. | `10s`, `120s`, `2m` etc. | \[Duration string]\[parse-duration] | `nil` (empty config, BIRD will use the default value of `120s`) | | localWorkloadPeeringIPV4 | Single virtual IPv4 address that all nodes will use for peering with local workloads when using `BGPPeer`'s `localWorkloadSelector`. A link local address is recommended but it must not be 169.254.0.1 or 169.254.0.2 because these are used internally by Calico Cloud. See the [guide](https://docs.tigera.io/calico-cloud/networking/configuring/bgp-to-workload) for details. | An IPv4 address. | string | None | | localWorkloadPeeringIPV6 | Single virtual IPv6 address that all nodes will use for peering with local workloads when using `BGPPeer`'s `localWorkloadSelector`. Must *not* be a link-local address (because Calico Cloud doesn't currently support the required "scope" suffix). See the [guide](https://docs.tigera.io/calico-cloud/networking/configuring/bgp-to-workload) for details. | An IPv6 address. | string | None | ### communities[​](#communities) | Field | Description | Accepted Values | Schema | | ----- | -------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------ | | name | Name or identifier for the community. This should be used in [prefixAdvertisements](#prefixadvertisements) to advertise the community value. | | string | | value | Standard or large BGP community value. | For standard community, value should be in `aa:nn` format, where both `aa` and `nn` are 16 bit integers. For large community, value should be `aa:nn:mm` format, where `aa`, `nn` and `mm` are all 32 bit integers. Where `aa` is an AS Number, `nn` and `mm` are per-AS identifier. | string | ### prefixAdvertisements[​](#prefixadvertisements) | Field | Description | Accepted Values | Schema | | ----------- | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------- | | cidr | CIDR for which properties should be advertised. | `cidr: XXX.XXX.XXX.XXX/XX` | string | | communities | BGP communities to be advertised. | Communities can be list of either community names already defined in [communities](#communities) or community value of format `aa:nn` or `aa:nn:mm`. For standard community, value should be in `aa:nn` format, where both `aa` and `nn` are 16 bit integers. For large community, value should be `aa:nn:mm` format, where `aa`, `nn` and `mm` are all 32 bit integers. Where `aa` is an AS Number, `nn` and `mm` are per-AS identifier. | List of string | ## Supported operations[​](#supported-operations) | Datastore type | Create | Delete | Delete (Global `default`) | Update | Get/List | Notes | | --------------------- | ------ | ------ | ------------------------- | ------ | -------- | ----- | | Kubernetes API server | Yes | Yes | No | Yes | Yes | | ### BGP peer A BGP peer resource (`BGPPeer`) represents a remote BGP peer with which the node(s) in a Calico Cloud cluster will peer. Configuring BGP peers allows you to peer a Calico Cloud network with your datacenter fabric (e.g. ToR). For more information on cluster layouts, see Calico Cloud's documentation on [Calico Cloud over IP fabrics](https://docs.tigera.io/calico-cloud/reference/architecture/design/l3-interconnect-fabric). For `kubectl` commands, the following case-insensitive aliases may be used to specify the resource type on the CLI: `bgppeer.projectcalico.org`, `bgppeers.projectcalico.org` as well as abbreviations such as `bgppeer.p` and `bgppeers.p`. ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: BGPPeer metadata: name: some.name spec: node: rack1-host1 peerIP: 192.168.1.1 asNumber: 63400 ``` ## BGP peer definition[​](#bgp-peer-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | | ----- | ------------------------------------------------------------------ | --------------------------------------------------- | ------ | | name | Unique name to describe this resource instance. Must be specified. | Alphanumeric string with optional `.`, `_`, or `-`. | string | ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | --------------------------- | ---------------------------------------------------------------------------- | | node | If specified, the scope is node level, otherwise the scope is global. | The hostname of the node to which this peer applies. | string | | | peerIP | The IP address of this peer and an optional port number. If port number is not set, and peer is Calico node with `listenPort` set, then `listenPort` is used. | Valid IPv4 or IPv6 address. If port number is set use, `IPv4:port` or `[IPv6]:port` format. | string | | | asNumber | The remote AS Number of the peer. | A valid AS Number, may be specified in dotted notation. | integer/string | | | nodeSelector | Selector for the nodes that should have this peering. When this is set, the `node` field must be empty. | | [selector](#selector) | | | peerSelector | Selector for the remote nodes to peer with. When this is set, the `peerIP` and `asNumber` fields must be empty. | | [selector](#selector) | | | localWorkloadSelector | Selector for the local workloads that the node should peer with. When this is set, the `peerSelector` and `peerIP` fields must be empty and the `localWorkloadPeeringIPV4` and/or `localWorkloadPeeringIPV6` fields in the `BGPConfiguration` resource must be configured. It is also important to configure appropriate import/export filters when using this feature. See the [guide](https://docs.tigera.io/calico-cloud/networking/configuring/bgp-to-workload) for details. | | [selector](#selectors) | | | keepOriginalNextHop | Maintain and forward the original next hop BGP route attribute to a specific Peer within a different AS. | | boolean | | | extensions | Additional mapping of keys and values. Used for setting values in custom BGP configurations. | valid strings for both keys and values | map | | | password | [BGP password](https://docs.tigera.io/calico-cloud/operations/comms/secure-bgp) for the peerings generated by this BGPPeer resource. | | [BGPPassword](#bgppassword) | `nil` (no password) | | sourceAddress | Specifies whether and how to configure a source address for the peerings generated by this BGPPeer resource. Default value "UseNodeIP" means to configure the node IP as the source address. "None" means not to configure a source address. | "UseNodeIP", "None" | string | "UseNodeIP" | | failureDetectionMode | Specifies whether and how to detect loss of connectivity on the peerings generated by this BGPPeer resource. Default value "None" means nothing beyond BGP's own (slow) hold timer. "BFDIfDirectlyConnected" means to use BFD when the peer is directly connected. | "None", "BFDIfDirectlyConnected" | string | "None" | | restartMode | Specifies restart behaviour to configure on the peerings generated by this BGPPeer resource. Default value "GracefulRestart" means traditional graceful restart. "LongLivedGracefulRestart" means LLGR according to draft-uttaro-idr-bgp-persistence-05. | "GracefulRestart", "LongLivedGracefulRestart" | string | "GracefulRestart" | | maxRestartTime | Restart time that is announced by BIRD in the BGP graceful restart capability and that specifies how long the neighbor would wait for the BGP session to re-establish after a restart before deleting stale routes. When specified, this is configured as the graceful restart timeout when `RestartMode` is "GracefulRestart", and as the LLGR stale time when `RestartMode` is "LongLivedGracefulRestart". When not specified, the BIRD defaults are used, which are 120s for "GracefulRestart" and 3600s for "LongLivedGracefulRestart". Note: extra care should be taken when changing this configuration, as it may break networking in your cluster. | | duration | None | | birdGatewayMode | Specifies the BIRD "gateway" mode, i.e. method for computing the immediate next hop for each received route, for peerings generated by this BGPPeer resource. Default value "Recursive" means "gateway recursive". "DirectIfDirectlyConnected" means to configure "gateway direct" when the peer is directly connected. | "Recursive", "DirectIfDirectlyConnected" | string | "Recursive" | | numAllowedLocalASNumbers | The number of local AS numbers to allow in the AS path for received routes. This disables BGP loop prevention and should only be used if necessary. | | integer | `nil` (BIRD will default to 0 meaning no change to loop prevention behavior) | | ttlSecurity | Enables the generalized TTL security mechanism (GTSM) which protects against spoofed packets by ignoring received packets with a smaller than expected TTL value. The provided value is the number of hops (edges) between the peers. | 0 - 255 | 8-bit integer | `nil` (results in BIRD configuration `ttl security off`) | | filters | List of names of [BGPFilter](https://docs.tigera.io/calico-cloud/reference/resources/bgpfilter) resources to apply to this peering. | \["my-bgp-filter-1","my-bgp-filter-2"] | List of strings | | | externalNetwork | Name of the external network to which this peer belongs. | - | string | | | reachableBy | Adds a static route that may be needed to connect to a peer. In some cases, not having a static route for BGP peering results in route flapping. By adding the address of the gateway that the peer is connected to, a static route is added to prevent route flapping. | The address of the gateway that the peer is connected to | string | | | nextHopMode | Specifies the method of calculating the next hop attribute for received routes. This replaces and expands the deprecated KeepOriginalNextHop field. Users should use this setting to control the next hop attribute for a BGP peer. When this is set, the value of the keepOriginalNextHop field is ignored. If neither keepOriginalNextHop or nextHopMode is specified, BGP's default behavior is used. Default value “Auto” means to apply BGP’s default behavior. "Self" means to configure "next hop self;" in "bird.cfg". "Keep" means to configure "next hop keep;" in "bird.cfg". | "Auto", "Self", "Keep" | string | "Auto" | | reversePeering | Specifies whether Calico automatically generates reverse peerings for nodes selected by the peerSelector. If set to Manual, a separate BGPPeer must be created for the reverse peering. | "Auto", "Manual" | string | "Auto" | > **SUCCESS:** The cluster-wide default local AS number used when speaking with a peer is controlled by the [BGPConfiguration resource](https://docs.tigera.io/calico-cloud/reference/resources/bgpconfig). That value can be overridden per-node by using the `bgp` field of the [node resource](https://docs.tigera.io/calico-cloud/reference/resources/node). ### BGPPassword[​](#bgppassword) > **SECONDARY:** BGP passwords must be 80 characters or fewer. If a password longer than that is configured, the BGP sessions with that password will fail to be established. | Field | Description | Schema | | ------------ | ------------------------------- | ----------------- | | secretKeyRef | Get the password from a secret. | [KeyRef](#keyref) | ### KeyRef[​](#keyref) KeyRef tells Calico Cloud where to get a BGP password. The referenced Kubernetes secret must be in the same namespace as the cnx-node pod. | Field | Description | Schema | | ----- | ------------------------- | ------ | | name | The name of the secret | string | | key | The key within the secret | string | > **WARNING:** Calico Cloud must be able to read the referenced secret. > > This means that the `calico-node` ServiceAccount must have permissions to `list, get, watch` the secret referenced in the KeyRef. > > In practice, this can be done by creating a Role (which allows to `list, get, watch` the secret) and a RoleBinding (which grants the Role's permission to the `calico-node` ServiceAccount). > > Example: > > ```yaml > --- > apiVersion: rbac.authorization.k8s.io/v1 > kind: Role > metadata: > name: bgp-passwords-reader > namespace: calico-system > rules: > - apiGroups: [""] > resources: ["secrets"] > resourceNames: ["bgp-passwords"] > verbs: ["list", "watch", "get"] > > --- > apiVersion: rbac.authorization.k8s.io/v1 > kind: RoleBinding > metadata: > name: calico-read-bgp-passwords > namespace: calico-system > roleRef: > apiGroup: rbac.authorization.k8s.io > kind: Role > name: bgp-passwords-reader > subjects: > - kind: ServiceAccount > name: calico-node > > --- > apiVersion: v1 > kind: Secret > metadata: > name: "bgp-passwords" > namespace: calico-system > data: > peer_a_pw: "base64-encoded Password for Peer A" > peer_b_pw: "base64-encoded Password for Peer B" > > --- > apiVersion: projectcalico.org/v3 > kind: BGPPeer > metadata: > name: "peer-a" > spec: > password: > secretKeyRef: > name: "bgp-passwords" > key: "peer_a_pw" > > --- > apiVersion: projectcalico.org/v3 > kind: BGPPeer > metadata: > name: "peer-b" > spec: > password: > secretKeyRef: > name: "bgp-passwords" > key: "peer_b_pw" > ``` ## Peer scopes[​](#peer-scopes) BGP Peers can exist at either global or node-specific scope. A peer's scope determines which `cnx-node`s will attempt to establish a BGP session with that peer. If `cnx-node` has a `listenPort` set in `BGPConfiguration`, it will be used in peering. ### Global peer[​](#global-peer) To assign a BGP peer a global scope, omit the `node` and `nodeSelector` fields. All nodes in the cluster will attempt to establish BGP connections with it ### Node-specific peer[​](#node-specific-peer) A BGP peer can also be node-specific. When the `node` field is included, only the specified node will peer with it. When the `nodeSelector` field is included, the nodes with labels that match that selector will peer with it. ## Supported operations[​](#supported-operations) | Datastore type | Create/Delete | Update | Get/List | Notes | | --------------------- | ------------- | ------ | -------- | ----- | | Kubernetes API server | Yes | Yes | Yes | | ## Selector[​](#selector) A label selector is an expression which either matches or does not match a resource based on its labels. Calico Cloud label selectors support a number of operators, which can be combined into larger expressions using the boolean operators and parentheses. | Expression | Meaning | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Logical operators** | | | `( )` | Matches if and only if `` matches. (Parentheses are used for grouping expressions.) | | `! ` | Matches if and only if `` does not match. **Tip:** `!` is a special character at the start of a YAML string, if you need to use `!` at the start of a YAML string, enclose the string in quotes. | | ` && ` | "And": matches if and only if both ``, and, `` matches | | ` \|\| ` | "Or": matches if and only if either ``, or, `` matches. | | **Match operators** | | | `all()` | Match all in-scope resources. To match *no* resources, combine this operator with `!` to form `!all()`. | | `global()` | Match all non-namespaced resources. Useful in a `namespaceSelector` to select global resources such as global network sets. | | `k == 'v'` | Matches resources with the label 'k' and value 'v'. | | `k != 'v'` | Matches resources without label 'k' or with label 'k' and value *not* equal to `v` | | `has(k)` | Matches resources with label 'k', independent of value. To match pods that do not have label `k`, combine this operator with `!` to form `!has(k)` | | `k in { 'v1', 'v2' }` | Matches resources with label 'k' and value in the given set | | `k not in { 'v1', 'v2' }` | Matches resources without label 'k' or with label 'k' and value *not* in the given set | | `k contains 's'` | Matches resources with label 'k' and value containing the substring 's' | | `k starts with 's'` | Matches resources with label 'k' and value starting with the substring 's' | | `k ends with 's'` | Matches resources with label 'k' and value ending with the substring 's' | Operators have the following precedence: - **Highest**: all the match operators - Parentheses `( ... )` - Negation with `!` - Conjunction with `&&` - **Lowest**: Disjunction with `||` For example, the expression ```text ! has(my-label) || my-label starts with 'prod' && role in {'frontend','business'} ``` Would be "bracketed" like this: ```text ((!(has(my-label)) || ((my-label starts with 'prod') && (role in {'frontend','business'})) ``` It would match: - Any resource that did not have label "my-label". - Any resource that both: - Has a value for `my-label` that starts with "prod", and, - Has a role label with value either "frontend", or "business". ### BGP Filter A BGP filter resource (`BGPFilter`) represents a way to control routes imported by and exported to BGP peers specified using a BGP peer resource (`BGPPeer`). The BGPFilter rules are applied sequentially: the `action` for the **first** rule that matches is executed immediately. If an address does not match any explicit BGP filter rule, the default action is `Accept`. In order for a BGPFilter to be used in a BGP peering, its `name` must be added to `filters` of the corresponding BGPPeer resource. For `kubectl` commands, the following case-sensitive aliases may be used to specify the resource type on the CLI: `bgpfilters.projectcalico.org` ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: BGPFilter metadata: name: my-filter spec: exportV4: - action: Accept matchOperator: In cidr: 77.0.0.0/16 source: RemotePeers - action: Reject interface: '*.calico' importV4: - action: Accept matchOperator: In cidr: 55.0.0.0/16 prefixLength: min: 30 - action: Reject matchOperator: NotIn cidr: 44.0.0.0/16 exportV6: - action: Reject source: RemotePeers - action: Reject interface: '*.calico' importV6: - action: Accept matchOperator: Equal cidr: 5000::0/64 - action: Reject ``` ## BGP filter definition[​](#bgp-filter-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | | ----- | ------------------------------------------------------------------ | --------------------------------------------------- | ------ | | name | Unique name to describe this resource instance. Must be specified. | Alphanumeric string with optional `.`, `_`, or `-`. | string | ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | -------- | ---------------------------------- | --------------- | ----------------------------------------- | ------- | | exportV4 | List of v4 CIDRs and export action | | [BGP Filter Rule v4](#bgp-filter-rule-v4) | | | importV4 | List of v4 CIDRs and import action | | [BGP Filter Rule v4](#bgp-filter-rule-v4) | | | exportV6 | List of v6 CIDRs and export action | | [BGP Filter Rule v6](#bgp-filter-rule-v6) | | | importV6 | List of v6 CIDRs and import action | | [BGP Filter Rule v6](#bgp-filter-rule-v6) | | ### BGP Filter Rule v4[​](#bgp-filter-rule-v4) | Field | Description | Accepted Values | Schema | Default | | ------------- | ----------------------------------------- | -------------------------------------------------------------------- | ------ | ------- | | cidr | IPv6 range | A valid IPv6 CIDR | string | | | prefixLength | [PrefixLength](#bgp-filter-prefix-length) | Valid integers between 0 and ipv4/6 max (32, 128) | | | | matchOperator | Method by which to match candidate routes | `In`, `NotIn`, `Equal`, `NotEqual` | string | | | source | Indicator of the source of route | `RemotePeers` means any route learned from other BGP peers | string | | | interface | String to match interface names | A valid pattern to match interfaces. "\*" can be used as a wildcard. | string | | | action | Action to be taken for this rule | `Accept` or `Reject` | string | | ### BGP Filter Rule v6[​](#bgp-filter-rule-v6) | Field | Description | Accepted Values | Schema | Default | | ------------- | ----------------------------------------- | -------------------------------------------------------------------- | ------ | ------- | | cidr | IPv6 range | A valid IPv6 CIDR | string | | | prefixLength | [PrefixLength](#bgp-filter-prefix-length) | Valid integers between 0 and ipv4/6 max (32, 128) | | | | matchOperator | Method by which to match candidate routes | `In`, `NotIn`, `Equal`, `NotEqual` | string | | | source | Indicator of the source of route | `RemotePeers` means any route learned from other BGP peers | string | | | interface | String to match interface names | A valid pattern to match interfaces. "\*" can be used as a wildcard. | string | | | action | Action to be taken for this rule | `Accept` or `Reject` | string | | ## Supported operations[​](#supported-operations) | Datastore type | Create/Delete | Update | Get/List | Notes | | --------------------- | ------------- | ------ | -------- | ----- | | Kubernetes API server | Yes | Yes | Yes | | ### Block affinity A block affinity resource (`BlockAffinity`) represents the affinity for an IPAM block. These are managed by Calico IPAM. ## Block affinity definition[​](#block-affinity-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | | ----- | ------------------------------------------------------------------ | --------------------------------------------------- | ------ | | name | Unique name to describe this resource instance. Must be specified. | Alphanumeric string with optional `.`, `_`, or `-`. | string | ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | ------- | -------------------------------------------------------------------------- | ----------------------------------- | ------- | ------- | | state | State of the affinity with regard to any referenced IPAM blocks. | confirmed, pending, pendingDeletion | string | | | node | The node that this affinity is assigned to. | The hostname of the node | string | | | cidr | The CIDR range this block affinity references. | A valid IPv4 or IPv6 CIDR. | string | | | deleted | When set to true, clients should treat this block as if it does not exist. | true, false | boolean | `false` | ## Supported operations[​](#supported-operations) | Datastore type | Create | Delete | Update | Get/List | Watch | | --------------------- | ------ | ------ | ------ | -------- | ----- | | etcdv3 | No | No | No | Yes | Yes | | Kubernetes API server | No | No | No | Yes | Yes | ### Calico node status A Calico node status resource (`CalicoNodeStatus`) represents a collection of status information for a node that Calico Cloud reports back to the user for use during troubleshooting. As of today, status of BGP agents, BGP sessions and routes exposed to BGP agents are collected from Linux nodes only. **Windows nodes are not supported at this time.** Calico node status resource is only valid when Calico Cloud BGP networking is in use. ### Notes[​](#notes) The updating of `CalicoNodeStatus` will have a small performance impact on CPU/Memory usage of the node as well as adding load to kubernetes apiserver. In our testing on a ten node, full mesh cluster, a `CalicoNodeStatus` resource was created for each node where the update interval was set to ten seconds. On each node, this resulted in an increase in CPU use of 5% of a vCPU and an increase of 4MB of memory. The control plane node recorded an increase in CPU usage of 5% of a vCPU for these 10 nodes. > **WARNING:** The implementation of `CalicoNodeStatus` is designed to handle a small number of nodes (less than 10 is recommended) reporting back status in the same time. If `CalicoNodeStatus` are created for a large number of nodes, and with short update interval, the kubernetes apiserver may become slower and less responsive. You should create `CalicoNodeStatus` for the node you are interested in and for debugging purpose only. `CalicoNodeStatus` resource should be deleted upon the completion of the debugging process. ## Sample YAML[​](#sample-yaml) To use this function, the user creates a CalicoNodeStatus object for the node, specifying the information to collect and the interval it should be collected at. This example collects information for node "my-kadm-node-0" with an update interval of 10 seconds. ```bash kubectl apply -f -< )` | Matches if and only if `` matches. (Parentheses are used for grouping expressions.) | | `! ` | Matches if and only if `` does not match. **Tip:** `!` is a special character at the start of a YAML string, if you need to use `!` at the start of a YAML string, enclose the string in quotes. | | ` && ` | "And": matches if and only if both ``, and, `` matches | | ` \|\| ` | "Or": matches if and only if either ``, or, `` matches. | | **Match operators** | | | `all()` | Match all in-scope resources. To match *no* resources, combine this operator with `!` to form `!all()`. | | `global()` | Match all non-namespaced resources. Useful in a `namespaceSelector` to select global resources such as global network sets. | | `k == 'v'` | Matches resources with the label 'k' and value 'v'. | | `k != 'v'` | Matches resources without label 'k' or with label 'k' and value *not* equal to `v` | | `has(k)` | Matches resources with label 'k', independent of value. To match pods that do not have label `k`, combine this operator with `!` to form `!has(k)` | | `k in { 'v1', 'v2' }` | Matches resources with label 'k' and value in the given set | | `k not in { 'v1', 'v2' }` | Matches resources without label 'k' or with label 'k' and value *not* in the given set | | `k contains 's'` | Matches resources with label 'k' and value containing the substring 's' | | `k starts with 's'` | Matches resources with label 'k' and value starting with the substring 's' | | `k ends with 's'` | Matches resources with label 'k' and value ending with the substring 's' | Operators have the following precedence: - **Highest**: all the match operators - Parentheses `( ... )` - Negation with `!` - Conjunction with `&&` - **Lowest**: Disjunction with `||` For example, the expression ```text ! has(my-label) || my-label starts with 'prod' && role in {'frontend','business'} ``` Would be "bracketed" like this: ```text ((!(has(my-label)) || ((my-label starts with 'prod') && (role in {'frontend','business'})) ``` It would match: - Any resource that did not have label "my-label". - Any resource that both: - Has a value for `my-label` that starts with "prod", and, - Has a role label with value either "frontend", or "business". Understanding scopes and the `all()` and `global()` operators: selectors have a scope of resources that they are matched against, which depends on the context in which they are used. For example: - The `nodeSelector` in an `IPPool` selects over `Node` resources. - The top-level selector in a `NetworkPolicy` selects over the workloads *in the same namespace* as the `NetworkPolicy`. - The top-level selector in a `GlobalNetworkPolicy` doesn't have the same restriction, it selects over all endpoints including namespaced `WorkloadEndpoint`s and non-namespaced `HostEndpoint`s. - The `namespaceSelector` in a `NetworkPolicy` (or `GlobalNetworkPolicy`) *rule* selects over the labels on namespaces rather than workloads. - The `namespaceSelector` determines the scope of the accompanying `selector` in the entity rule. If no `namespaceSelector` is present then the rule's `selector` matches the default scope for that type of policy. (This is the same namespace for `NetworkPolicy` and all endpoints/network sets for `GlobalNetworkPolicy`) - The `global()` operator can be used (only) in a `namespaceSelector` to change the scope of the main `selector` to include non-namespaced resources such as [GlobalNetworkSet](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkset). This allows namespaced `NetworkPolicy` resources to refer to global non-namespaced resources, which would otherwise be impossible. ### StringValueMatch[​](#stringvaluematch) A string values match does a match or negative match on a target against a set of values. | Field | Description | Accepted Values | Schema | Default | | -------- | ------------------------------------------------ | --------------------- | --------------- | ------- | | operator | Match operator to use against the list of values | `IsOneOf`, `IsNoneOf` | string | | | values | List of values to match against. | | list of strings | | ### TimeMatch[​](#timematch) A time match does a match or negative match against a time or duration. Duration is the length of time into the past from the current time to match against. As an example, ```yaml operator: "gt" duration: days: 3 ``` is false for 4 days prior to the current date time, and true for 2 days prior to current date time. Time is the absolute time to match a given time against. As an example, ```yaml operator: "gt" time: "2022-01-02T0:00:00Z" ``` is false for "2022-01-01T0:00:00Z", and true for "2022-01-03T0:00:00Z". | Field | Description | Accepted Values | Schema | Default | | -------- | --------------------------------------------------- | --------------- | ------------------------------------------------ | ------- | | operator | Match operator to use against the duration or time. | `gt`, `lt` | string | | | duration | The duration that this operator matches within. | | [Duration](#duration) | | | time | List of values to match against. | | String of the format "2006-01-02T15:04:05Z07:00" | | ### Duration[​](#duration) | Field | Description | Accepted Values | Schema | Default | | ------ | -------------------------------------------- | --------------- | ------- | ------- | | days | Number of days past from the current time. | | integer | | | months | Number of months past from the current time. | | integer | | | years | Number of years past from the current time. | | integer | | ## Supported operations[​](#supported-operations) | Datastore type | Create/Delete | Update | Get/List | Notes | | ------------------------ | ------------- | ------ | -------- | ----- | | Kubernetes API datastore | Yes | Yes | Yes | | ### Compliance reports (deprecated) ## [📄️ Compliance reports (deprecated)](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/overview) [Schedule reports and configure report scope.](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/overview) ## [📄️ Inventory report](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/inventory) [API for this resource.](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/inventory) ## [📄️ Network Access report](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/network-access) [API for this resource.](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/network-access) ## [📄️ Policy audit report](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/policy-audit) [API for this resource.](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/policy-audit) ## [📄️ CIS benchmark report](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/cis-benchmark) [API for this resource.](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/cis-benchmark) ### Compliance reports (deprecated) The Calico Cloud compliance reporting feature provides the following compliance reports: - [Inventory](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/inventory) - [Network Access](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/network-access) - [Policy Audit](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/policy-audit) - [CIS Benchmark](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/cis-benchmark) Create a [`GlobalReport`](https://docs.tigera.io/calico-cloud/reference/resources/globalreport) resource to automatically schedule report generation, and specify the report scope (resources to include in the report). ## Concepts[​](#concepts) ### In-scope asset[​](#in-scope-asset) An asset (Pod or HostEndpoint) is flagged as in-scope by endpoint labels, namespace and/or namespace labels, and service account and/or service account labels. *How this applies to the report*: The report includes all resources that were in-scope at any point during the report interval. The resource is included when it is first flagged as in-scope according to the configured label selector and name selections. The resource is included even if the resource is deleted or goes out-of-scope before the end of the report interval. ### Ingress protected[​](#ingress-protected) An endpoint is ingress protected if it has at least one Ingress policy that is applied to it. A service is ingress protected if all of the inscope endpoints within that service are ingress protected. A namespace is ingress protected if all of the inscope endpoints within that namespace are ingress protected. *How this applies to the report*: An endpoint is ingress protected only if it was ingress protected throughout the entire report interval. ### Egress protected[​](#egress-protected) As per ingress, but with egress policy rules. Note that egress statistics are not obtained for services. ### Allows ingress traffic from another namespace[​](#allows-ingress-traffic-from-another-namespace) An endpoint is flagged as allowing ingress traffic from another namespace if it has one or more policies that apply to it with an ingress allow rule that: - has an explicit namespace selector configured, or - has no source selector or source CIDR configured, or - (for GlobalNetworkPolicy) has no source CIDR. A service is flagged as allowing ingress traffic from another namespace if any of the inscope endpoints within that service are flagged. A namespace is flagged as allowing ingress traffic from another namespace if all of the inscope endpoints within that namespace are flagged. *How this applies to the report*: An endpoint is flagged as allowing ingress traffic from another namespace if it was flagged at any time during the report interval. ### Allows egress traffic to another namespace[​](#allows-egress-traffic-to-another-namespace) As per ingress, but with egress policy rules and destination selector/CIDR. Note that egress statistics are not obtained for services. ### Allows ingress traffic from the internet[​](#allows-ingress-traffic-from-the-internet) An endpoint is flagged as allowing ingress traffic from the internet if it has one or more policies that apply to it with an ingress allow rule that: - has no source selector or source CIDR configured, or - has a source CIDR in the non-private IP ranges and has no source selector, or - has a source selector that matches one or more NetworkSets that contain at least one non-private IP. A service is flagged as allowing ingress traffic from the internet if any of the inscope endpoints within that service are flagged. A namespace is flagged as allowing ingress traffic from the internet if all of the inscope endpoints within that namespace are flagged. *How this applies to the report*: An endpoint is flagged as allowing ingress traffic from the internet if it was flagged as such at any time during the report interval. ### Allows egress traffic to the internet[​](#allows-egress-traffic-to-the-internet) As per ingress, but with egress policy rules and destination selector/CIDR. Note that egress statistics are not obtained for services. ### Envoy enabled[​](#envoy-enabled) An endpoint is flagged as Envoy Enabled if the associated Pod Spec and Annotations indicate that an Istio init and main container are deployed in the Pod. Provided Istio is appropriately configured on the cluster, this can be extrapolated to be indication of whether mTLS is enabled for the endpoint. A service is flagged as Envoy enabled if all of the inscope endpoints within that service are flagged. A namespace is flagged as Envoy enabled if all of the inscope endpoints within that namespace are flagged. *How this applies to the report*: An endpoint is flagged as Envoy enabled if it was flagged as such throughout the entire report interval. ### Inventory report To create an Inventory report, create a [`GlobalReport`](https://docs.tigera.io/calico-cloud/reference/resources/globalreport) with the `reportType` set to `inventory`. The following sample command creates a GlobalReport that results in a daily inventory report for endpoints in the `public` namespace. ```bash kubectl apply -f - << EOF apiVersion: projectcalico.org/v3 kind: GlobalReport metadata: name: daily-public-inventory-report labels: deployment: production spec: reportType: inventory endpoints: namespaces: names: - public schedule: 0 0 * * * EOF ``` ## Downloadable reports[​](#downloadable-reports) ### summary.csv[​](#summarycsv) A summary CSV file that includes details about the report parameters and the top level counts. | Heading | Description | Format | | ----------------------------- | ----------------------------------------------------------------------------------------------------------- | ------------------------------------------- | | startTime | The report interval start time. | RFC3339 string | | endTime | The report interval end time. | RFC3339 string | | endpointSelector | The endpoint selector used to restrict in-scope endpoints by endpoint label selection. | selector string | | namespaceNames | The set of namespace names used to restrict in-scope endpoints by namespace. | ";" separated list of namespace names | | namespaceSelector | The namespace selector used to restrict in-scope endpoints by namespace label selection. | selector string | | serviceAccountNames | The set of service account names used to restrict in-scope endpoints by service account. | ";" separated list of service account names | | serviceAccountSelectors | The service account selector used to restrict in-scope endpoints by service account label selection. | selector string | | endpointsNumInScope | The number of enumerated endpoints that are in-scope according to the requested endpoint selection options. | number | | endpointsNumIngressProtected | The number of in-scope endpoints that were always ingress protected during the report interval. | number | | endpointsNumEgressProtected | The number of in-scope endpoints that were always egress protected during the report interval. | number | | namespacesNumInScope | The number of namespaces containing in-scope endpoints. | number | | namespacesNumIngressProtected | The number of namespaces whose in-scope endpoints were always ingress protected during the report interval. | number | | namespacesNumEgressProtected | The number of namespaces whose in-scope endpoints were always egress protected during the report interval. | number | | serviceAccountsNumInScope | The number of service accounts associated with in-scope endpoints. | number | ### endpoints.csv[​](#endpointscsv) An endpoints CSV file that includes per-endpoint information. | Heading | Description | Format | | ---------------- | --------------------------------------------------------------------------------------------- | ----------------------------------- | | endpoint | The name of the endpoint. | string | | ingressProtected | Whether the endpoint was always ingress protected during the report interval. | bool | | egressProtected | Whether the endpoint was always egress protected during the report interval. | bool | | envoyEnabled | Whether the endpoint was always Envoy enabled during the report interval. | bool | | appliedPolicies | The full set of policies that applied to the endpoint at any time during the report interval. | ";" separated list of policy names | | services | The full set of services that included this endpoint at any time during the report interval. | ";" separated list of service names | ### namespaces.csv[​](#namespacescsv) A namespaces CSV file that includes per-namespace information. | Heading | Description | Format | | ---------------- | ------------------------------------------------------------------------------------------------------------- | ------ | | namespace | The name of the namespace. | string | | ingressProtected | Whether all in-scope endpoints within the namespace were always ingress protected during the report interval. | bool | | egressProtected | Whether all in-scope endpoints within the namespace were always egress protected during the report interval. | bool | | envoyEnabled | Whether all in-scope endpoints within the namespace were always Envoy enabled during the report interval. | bool | ### services.csv[​](#servicescsv) A services CSV file that includes per-service information. | Heading | Description | Format | | ---------------- | ---------------------------------------------------------------------------------------------------------------- | ------ | | service | The name of the service. | string | | ingressProtected | Whether all in-scope endpoints that are in the service were always ingress protected during the report interval. | bool | | envoyEnabled | Whether all in-scope endpoints that are in the service were always Envoy enabled during the report interval. | bool | ### Network Access report To create an Inventory report, create a [`GlobalReport`](https://docs.tigera.io/calico-cloud/reference/resources/globalreport) with the `reportType` set to `network-access`. The following sample command creates a GlobalReport that results in a daily network access report for endpoints in the `public` namespace. ```bash kubectl apply -f - << EOF apiVersion: projectcalico.org/v3 kind: GlobalReport metadata: name: daily-public-network-access-report labels: deployment: production spec: reportType: network-access endpoints: namespaces: names: - public schedule: 0 0 * * * EOF ``` > **SECONDARY:** There is a known issue that audit logs do not contain deletion events for resources that were deleted implicitly as part of a namespace deletion event. Currently, this means policies and pods that have been deleted in this way may still appear in the reports that cover any period within the next day. ## Downloadable reports[​](#downloadable-reports) ### summary.csv[​](#summarycsv) A summary CSV file that includes details about the report parameters and the top level counts. | Heading | Description | Format | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | | startTime | The report interval start time. | RFC3339 string | | endTime | The report interval end time. | RFC3339 string | | endpointSelector | The endpoint selector used to restrict in-scope endpoints by endpoint label selection. | selector string | | namespaceNames | The set of namespace names used to restrict in-scope endpoints by namespace. | ";" separated list of namespace names | | namespaceSelector | The namespace selector used to restrict in-scope endpoints by namespace label selection. | selector string | | serviceAccountNames | The set of service account names used to restrict in-scope endpoints by service account. | ";" separated list of service account names | | serviceAccountSelectors | The service account selector used to restrict in-scope endpoints by service account label selection. | selector string | | endpointsNumIngressProtected | The number of in-scope endpoints that were always ingress protected during the report interval. | number | | endpointsNumEgressProtected | The number of in-scope endpoints that were always egress protected during the report interval. | number | | endpointsNumIngressUnprotected | The number of in-scope endpoints that were ingress unprotected at any point during the report interval. | number | | endpointsNumEgressUnprotected | The number of in-scope endpoints that were egress unprotected at any point during the report interval. | number | | endpointsNumIngressFromInternet | The number of in-scope endpoints that allowed ingress traffic from the public internet at any point during the report interval. | number | | endpointsNumEgressToInternet | The number of in-scope endpoints that allowed egress traffic to the public internet at any point during the report interval. | number | | endpointsNumIngressFromOtherNamespace | The number of in-scope endpoints that allowed ingress traffic from another namespace at any point during the report interval. | number | | endpointsNumEgressToOtherNamespace | The number of in-scope endpoints that allowed egress traffic to another namespace at any point during the report interval. | number | | endpointsNumEnvoyEnabled | The number of in-scope endpoints that were always Envoy enabled during the report interval. | number | ### endpoints.csv[​](#endpointscsv) An endpoints CSV file that includes per-endpoint information. | Heading | Description | Format | | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | ----------------------------------- | | endpoint | The name of the endpoint. | string | | ingressProtected | Whether the endpoint was always ingress protected during the report interval. | bool | | egressProtected | Whether the endpoint was always egress protected during the report interval. | bool | | ingressFromInternet | Whether the endpoint allowed ingress traffic from the public internet at any point during the report interval. | number | | egressToInternet | Whether the endpoint allowed egress traffic to the public internet at any point during the report interval. | number | | ingressFromOtherNamespace | Whether the endpoint allowed ingress traffic from another namespace at any point during the report interval. | number | | egressToOtherNamespace | Whether the endpoint allowed egress traffic to another namespace at any point during the report interval. | number | | envoyEnabled | Whether the endpoint was always Envoy enabled during the report interval. | bool | | appliedPolicies | The full set of policies that applied to the endpoint at any time during the report interval. | ";" separated list of policy names | | services | The full set of services that included this endpoint at any time during the report interval. | ";" separated list of service names | | trafficAggregationPrefix\* | The flow log aggregation prefix. | string | | endpointsGeneratingTrafficToThisEndpoint\* | The set of endpoints that were generating traffic to this endpoint. | ";" separated list of service names | | endpointsReceivingTrafficFromThisEndpoint\* | The set of endpoints that this endpoint is generating traffic to. | ";" separated list of service names | \* Traffic data is determined from flow logs. By default, Calico Cloud aggregates flow logs so that flows to and from pods in the same replica set are summarized if the flows are accepted. (Denied flows are not aggregated this way by default). This means that the per-endpoint traffic details do not refer specifically to that endpoint, but rather the set of endpoints specified by the trafficAggregationPrefix. If you want per-endpoint detail you should turn down the level of aggregation. To do so, set the value of `flowLogsFileAggregationKindForAllowed` to 1 using a [FelixConfiguration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig) ### Policy audit report To create a Policy Audit report, create a [`GlobalReport`](https://docs.tigera.io/calico-cloud/reference/resources/globalreport) with the `reportType` set to `policy-audit`. The following sample command creates a GlobalReport that results in a daily policy audit report for policies that are applied to endpoints in the `public` namespace. ```bash kubectl apply -f - << EOF apiVersion: projectcalico.org/v3 kind: GlobalReport metadata: name: daily-public-policy-audit-report labels: deployment: production spec: reportType: policy-audit endpoints: namespaces: names: - public schedule: 0 0 * * * EOF ``` ## Downloadable reports[​](#downloadable-reports) ### summary.csv[​](#summarycsv) A summary CSV file that includes details about the report parameters and the top level counts. | Heading | Description | Format | | ----------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------------- | | startTime | The report interval start time. | RFC3339 string | | endTime | The report interval end time. | RFC3339 string | | endpointSelector | The endpoint selector used to restrict in-scope endpoints by endpoint label selection. | selector string | | namespaceNames | The set of namespace names used to restrict in-scope endpoints by namespace. | ";" separated list of namespace names | | namespaceSelector | The namespace selector used to restrict in-scope endpoints by namespace label selection. | selector string | | serviceAccountNames | The set of service account names used to restrict in-scope endpoints by service account. | ";" separated list of service account names | | serviceAccountSelectors | The service account selector used to restrict in-scope endpoints by service account label selection. | selector string | | numCreatedPolicies | The number of policies that apply to in-scope endpoints that were created during the report interval. | number | | numModifiedPolicies | The number of policies that apply to in-scope endpoints that were modified during the report interval. | number | | numDeletedPolicies | The number of policies that apply to in-scope endpoints that were deleted during the report interval. | number | ### events.json[​](#eventsjson) Events formatted in JSON. ### events.yaml[​](#eventsyaml) Events formatted in YAML. ### CIS benchmark report To create a CIS benchmark report, create a `GlobalReport` with the `reportType` set to `cis-benchmark`. The following sample command uses a GlobalReport to create a daily CIS benchmark report that run on all the nodes. ```text kubectl apply -f - << EOF apiVersion: projectcalico.org/v3 kind: GlobalReport metadata: name: daily-cis-benchmark-report spec: reportType: cis-benchmark schedule: 0 0 * * * EOF ``` ## Openshift[​](#openshift) While there is no extra setup configuration required by the user to generate a benchmark report for Openshift, the result sets will be different than a report generated for regular Kubernetes clusters. Use the [Openshift Container Platform Security Guide](https://static.open-scap.org/ssg-guides/ssg-ocp4-guide-index.html) to cross-reference the benchmark results. ## Downloadable reports[​](#downloadable-reports) ## total-summary.csv[​](#total-summarycsv) A textual representation of the dashboard. | Heading | Description | Format | | ---------------------- | ----------------------------------------------------------------- | -------------- | | startTime | The report interval start time. | RFC3339 string | | endTime | The report interval start time. | RFC3339 string | | type | The type of benchmark report | string | | hiPercentageThreshold | The percentage of passing tests required to rate a node as high | int | | medPercentageThreshold | The percentage of passing tests required to rate a node as medium | int | | hiNodeCount | The number of nodes rated as high | int | | medNodeCount | The number of nodes rated as medium | int | | lowNodeCount | The number of nodes rated as low | int | ## node-summary.csv[​](#node-summarycsv) A .csv file of test result summaries per node. | Heading | Description | Format | | ------------ | ---------------------------------------------------------------------------------- | ------ | | node | The name of the node. | string | | version | The version of the platform. | string | | status | The rating of the node based on percentage of tests passing. | string | | testsPassing | The number of tests passing. | int | | testsFailing | The number of tests failing. | int | | testsUnknown | The number of tests whose results are undetermined due to automation restrictions. | int | | testsTotal | The total number of tests executed. | int | ### failed-tests.csv[​](#failed-testscsv) A .csv file of tests that have failed. | Heading | Description | Format | | --------- | -------------------------------------------------------------------------------------- | ------ | | nodeName | Node where the test is executed. | string | | testIndex | Index of the test on the Kubernetes CIS benchmark. | string | | status | Test results: PASS, FAIL, INFO. | string | | scored | Indicates whether the Kubernetes CIS benchmark counts this test towards their scoring. | string | ### all-tests.csv[​](#all-testscsv) A .csv file with tests that were executed on all nodes. Format remains the same as above. ### Deep packet inspection A deep packet inspection resource (`DeepPacketInspection`) represents live network traffic monitor for malicious activities by analyzing header and payload of the packet using specific rules. Malicious activities are added to the “Alerts” page in the Calico Cloud web console. For `kubectl` [commands](https://kubernetes.io/docs/reference/kubectl/overview/), the following case-insensitive aliases can be used to specify the resource type on the CLI: `deeppacketinspection`,`deeppacketinspections`, `deeppacketinspection.projectcalico.org`, `deeppacketinspections.projectcalico.org` as well as abbreviations such as `deeppacketinspection.p` and `deeppacketinspections.p`. ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: DeepPacketInspection metadata: name: sample-dpi namespace: sample-namespace spec: selector: k8s-app == "sample-app" ``` ## DeepPacketInspection definition[​](#deeppacketinspection-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | Default | | --------- | ------------------------------------------------------------------ | --------------------------------------------------- | ------ | --------- | | name | The name of the deep packet inspection. Required. | Alphanumeric string with optional `.`, `_`, or `-`. | string | | | namespace | Namespace provides an additional qualification to a resource name. | | string | "default" | ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | -------- | ------------------------------------------------------------------- | --------------- | --------------------- | ------- | | selector | Selects the endpoints to which this deep packet inspection applies. | | [selector](#selector) | | ### Status[​](#status) | Field | Description | | ----- | ------------------------ | | nodes | List of [Nodes](#nodes). | ### Nodes[​](#nodes) | Field | Description | | --------------- | -------------------------------------------- | | node | Name of the node that generated this status. | | active | [Active](#active) status. | | errorConditions | List of [errors](#error-conditions). | ### Active[​](#active) | Field | Description | | ----------- | ------------------------------------------------------------ | | success | Whether the deep packet inspection is active on the backend. | | lastUpdated | Time when the [active](#active) field was updated. | ### Error Conditions[​](#error-conditions) | Field | Description | | ----------- | ------------------------------------------------------------------- | | message | Errors preventing deep packet inspection from running successfully. | | lastUpdated | Time when the [error](#error-conditions) was updated. | ### Selector[​](#selector) A label selector is an expression which either matches or does not match a resource based on its labels. Calico Cloud label selectors support a number of operators, which can be combined into larger expressions using the boolean operators and parentheses. | Expression | Meaning | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Logical operators** | | | `( )` | Matches if and only if `` matches. (Parentheses are used for grouping expressions.) | | `! ` | Matches if and only if `` does not match. **Tip:** `!` is a special character at the start of a YAML string, if you need to use `!` at the start of a YAML string, enclose the string in quotes. | | ` && ` | "And": matches if and only if both ``, and, `` matches | | ` \|\| ` | "Or": matches if and only if either ``, or, `` matches. | | **Match operators** | | | `all()` | Match all in-scope resources. To match *no* resources, combine this operator with `!` to form `!all()`. | | `global()` | Match all non-namespaced resources. Useful in a `namespaceSelector` to select global resources such as global network sets. | | `k == 'v'` | Matches resources with the label 'k' and value 'v'. | | `k != 'v'` | Matches resources without label 'k' or with label 'k' and value *not* equal to `v` | | `has(k)` | Matches resources with label 'k', independent of value. To match pods that do not have label `k`, combine this operator with `!` to form `!has(k)` | | `k in { 'v1', 'v2' }` | Matches resources with label 'k' and value in the given set | | `k not in { 'v1', 'v2' }` | Matches resources without label 'k' or with label 'k' and value *not* in the given set | | `k contains 's'` | Matches resources with label 'k' and value containing the substring 's' | | `k starts with 's'` | Matches resources with label 'k' and value starting with the substring 's' | | `k ends with 's'` | Matches resources with label 'k' and value ending with the substring 's' | Operators have the following precedence: - **Highest**: all the match operators - Parentheses `( ... )` - Negation with `!` - Conjunction with `&&` - **Lowest**: Disjunction with `||` For example, the expression ```text ! has(my-label) || my-label starts with 'prod' && role in {'frontend','business'} ``` Would be "bracketed" like this: ```text ((!(has(my-label)) || ((my-label starts with 'prod') && (role in {'frontend','business'})) ``` It would match: - Any resource that did not have label "my-label". - Any resource that both: - Has a value for `my-label` that starts with "prod", and, - Has a role label with value either "frontend", or "business". ### Felix configuration A Felix configuration resource (`FelixConfiguration`) represents Felix configuration options for the cluster. For `kubectl` commands, the following case-insensitive aliases may be used to specify the resource type on the CLI: `felixconfiguration.projectcalico.org`, `felixconfigurations.projectcalico.org` as well as abbreviations such as `felixconfiguration.p` and `felixconfigurations.p`. See [Configuring Felix](https://docs.tigera.io/calico-cloud/reference/component-resources/node/felix/configuration) for more details. ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: FelixConfiguration metadata: name: default spec: ipv6Support: false ipipMTU: 1400 chainInsertMode: Append ``` ## Felix configuration definition[​](#felix-configuration-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | | ----- | --------------------------------------------------------- | --------------------------------------------------- | ------ | | name | Unique name to describe this resource instance. Required. | Alphanumeric string with optional `.`, `_`, or `-`. | string | - Calico Cloud automatically creates a resource named `default` containing the global default configuration settings for Felix. - The resources with the name `node.` contain the node-specific overrides, and will be applied to the node ``. When deleting a node the FelixConfiguration resource associated with the node will also be deleted. ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | awsSrcDstCheck | Controls automatically setting [source-destination-check](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_NAT_Instance.html#EIP_Disable_SrcDestCheck) on an AWS EC2 instance running Felix. Setting the value to `Enable` will set the check value in the instance description to `true`. For `Disable`, the check value will be `false`. Setting must be `Disable` if you want the EC2 instance to process traffic not matching the host interface IP address. For example, EKS cluster using Calico CNI with `VXLANMode=CrossSubnet`. Check [IAM role and profile configuration](#aws-iam-rolepolicy-for-source-destination-check-configuration) for setting the necessary permission for this setting to work. | DoNothing, Enable, Disable | string | `DoNothing` | | awsSecondaryIPSupport | Controls whether Felix will create secondary AWS ENIs for AWS-backed IP pools. This feature is documented in the [egress gateways on AWS guide](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-aws). Should only be enabled on AWS. | `Enabled`, `EnabledENIPerWorkload`, `Disabled` | string | `Disabled` | | awsSecondaryIPRoutingRulePriority | Controls the priority of the policy-based routing rules used to implement AWS-backed IP addresses. Should only be changed to avoid conflicts if your nodes have additional policy based routing rules. | 0-4294967295 | int | 101 | | awsRequestTimeout | Timeout used for communicating with the AWS API. | `5s`, `10s`, `1m` etc. | duration | `30s` | | dropActionOverride | Controls what happens to each packet that is denied by the current Calico Cloud policy. Normally the `Drop` or `LogAndDrop` value should be used. However when experimenting or debugging a scenario that is not behaving as you expect, the `Accept` and `LogAndAccept` values can be useful: then the packet will be still be allowed through. When one of the `LogAnd...` values is set, each denied packet is logged in syslog.\* | `Drop`, `Accept`, `LogAndDrop`, `LogAndAccept` | string | `Drop` | | chainInsertMode | Controls whether Felix hooks the kernel's top-level iptables chains by inserting a rule at the top of the chain or by appending a rule at the bottom. `Insert` is the safe default since it prevents Calico Cloud's rules from being bypassed. If you switch to `Append` mode, be sure that the other rules in the chains signal acceptance by falling through to the Calico Cloud rules, otherwise the Calico Cloud policy will be bypassed. | `Insert`, `Append` | string | `Insert` | | healthTimeoutOverrides | A list of overrides for Felix's internal liveness/readiness timeouts. | see [below](#health-timeout-overrides) | List of `HealthTimeoutOverride` objects | `[]` | | dataplaneWatchdogTimeout | Deprecated, use `healthTimeoutOverrides` instead. Timeout before the main data plane goroutine is determined to have hung and Felix will report non-live and non-ready. Can be increased if the liveness check incorrectly fails (for example if Felix is running slowly on a heavily loaded system). | `90s`, `120s`, `10m` etc. | duration | `90s` | | defaultEndpointToHostAction | This parameter controls what happens to traffic that goes from a workload endpoint to the host itself (after the traffic hits the endpoint egress policy). By default Calico Cloud blocks traffic from workload endpoints to the host itself with an iptables "DROP" action. If you want to allow some or all traffic from endpoint to host, set this parameter to `Return` or `Accept`. Use `Return` if you have your own rules in the iptables "INPUT" chain; Calico Cloud will insert its rules at the top of that chain, then `Return` packets to the "INPUT" chain once it has completed processing workload endpoint egress policy. Use `Accept` to unconditionally accept packets from workloads after processing workload endpoint egress policy. | Drop, Return, Accept | string | `Drop` | | deviceRouteSourceAddress | IPv4 address to set as the source hint for routes programmed by Felix. When not set the source address for local traffic from host to workload will be determined by the kernel. | IPv4 | string | `""` | | deviceRouteSourceAddressIPv6 | IPv6 address to set as the source hint for routes programmed by Felix. When not set the source address for local traffic from host to workload will be determined by the kernel. | IPv6 | string | `""` | | deviceRouteProtocol | This defines the route protocol added to programmed device routes. | Protocol | int | RTPROT\_BOOT | | externalNodesCIDRList | A comma-delimited list of CIDRs of external non-calico nodes that can source tunnel traffic for acceptance by calico-nodes. | IPv4 | string | `""` | | failsafeInboundHostPorts | UDP/TCP/SCTP protocol/cidr/port groupings that Felix will allow incoming traffic to host endpoints on irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. The default value allows SSH access, etcd, BGP, DHCP and the Kubernetes API. | | List of [ProtoPort](#protoport) | `- protocol: tcp   port: 22 - protocol: udp   port: 68 - protocol: tcp   port: 179 - protocol: tcp   port: 2379 - protocol: tcp   port: 2380 - protocol: tcp   port: 5473 - protocol: tcp   port: 6443 - protocol: tcp   port: 6666 - protocol: tcp   port: 6667` | | failsafeOutboundHostPorts | UDP/TCP/SCTP protocol/port groupings that Felix will allow outgoing traffic from host endpoints to irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. The default value opens etcd's standard ports to ensure that Felix does not get cut off from etcd as well as allowing DHCP, DNS, BGP and the Kubernetes API. | | List of [ProtoPort](#protoport) | `- protocol: udp   port: 53 - protocol: udp   port: 67 - protocol: tcp   port: 179 - protocol: tcp   port: 2379 - protocol: tcp   port: 2380 - protocol: tcp   port: 5473 - protocol: tcp   port: 6443 - protocol: tcp   port: 6666 - protocol: tcp   port: 6667` | | featureDetectOverride | Is used to override the feature detection. Values are specified in a comma separated list with no spaces, example; "SNATFullyRandom=true,MASQFullyRandom=false,RestoreSupportsLock=". "true" or "false" will force the feature, empty or omitted values are auto-detected. | string | string | `""` | | genericXDPEnabled | When enabled, Felix can fallback to the non-optimized `generic` XDP mode. This should only be used for testing since it doesn't improve performance over the non-XDP mode. | true,false | boolean | `false` | | interfaceExclude | A comma-separated list of interface names that should be excluded when Felix is resolving host endpoints. The default value ensures that Felix ignores Kubernetes' internal `kube-ipvs0` device. If you want to exclude multiple interface names using a single value, the list supports regular expressions. For regular expressions you must wrap the value with `/`. For example having values `/^kube/,veth1` will exclude all interfaces that begin with `kube` and also the interface `veth1`. | string | string | `kube-ipvs0` | | interfacePrefix | The interface name prefix that identifies workload endpoints and so distinguishes them from host endpoint interfaces. Note: in environments other than bare metal, the orchestrators configure this appropriately. For example our Kubernetes and Docker integrations set the 'cali' value, and our OpenStack integration sets the 'tap' value. | string | string | `cali` | | ipipEnabled | Optional, you shouldn't need to change this setting as Felix calculates if IPIP should be enabled based on the existing IP Pools. When set, this overrides whether Felix should configure an IPinIP interface on the host. When explicitly disabled in FelixConfiguration, Felix will not clean up addresses from the `tunl0` interface (use this if you need to add addresses to that interface and don't want to have them removed). | `true`, `false`, unset | optional boolean | unset | | ipipMTU | The MTU to set on the tunnel device. Zero value means auto-detect. See [Configuring MTU](https://docs.tigera.io/calico-cloud/networking/configuring/mtu) | int | int | `0` | | ipsetsRefreshInterval | Period at which Felix re-checks the IP sets in the data plane to ensure that no other process has accidentally broken Calico Cloud's rules. Set to 0 to disable IP sets refresh. | `5s`, `10s`, `1m` etc. | duration | `10s` | | iptablesFilterAllowAction | This parameter controls what happens to traffic that is accepted by a Felix policy chain in the iptables filter table (i.e. a normal policy chain). The default will immediately `Accept` the traffic. Use `Return` to send the traffic back up to the system chains for further processing. | Accept, Return | string | `Accept` | | iptablesBackend | This parameter controls which variant of iptables Felix uses. If using Felix on a system that uses the netfilter-backed iptables binaries, set this to `nft`. | Legacy, nft | string | automatic detection | | iptablesLockFilePath | Location of the iptables lock file. You may need to change this if the lock file is not in its standard location (for example if you have mapped it into Felix's container at a different path). | string | string | `/run/xtables.lock` | | iptablesLockProbeInterval | Time that Felix will wait between attempts to acquire the iptables lock if it is not available. Lower values make Felix more responsive when the lock is contended, but use more CPU. | `5s`, `10s`, `1m` etc. | duration | `50ms` | | iptablesLockTimeout | Time that Felix will wait for the iptables lock, or 0, to disable. To use this feature, Felix must share the iptables lock file with all other processes that also take the lock. When running Felix inside a container, this requires the /run directory of the host to be mounted into the cnx-node or calico/felix container. | `5s`, `10s`, `1m` etc. | duration | `0` (Disabled) | | iptablesMangleAllowAction | This parameter controls what happens to traffic that is accepted by a Felix policy chain in the iptables mangle table (i.e. a pre-DNAT policy chain). The default will immediately `Accept` the traffic. Use `Return` to send the traffic back up to the system chains for further processing. | `Accept`, `Return` | string | `Accept` | | iptablesMarkMask | Mask that Felix selects its IPTables Mark bits from. Should be a 32 bit hexadecimal number with at least 8 bits set, none of which clash with any other mark bits in use on the system. | netmask | netmask | `0xffff0000` | | iptablesNATOutgoingInterfaceFilter | This parameter can be used to limit the host interfaces on which Calico will apply SNAT to traffic leaving a Calico IPAM pool with "NAT outgoing" enabled. This can be useful if you have a main data interface, where traffic should be SNATted and a secondary device (such as the docker bridge) which is local to the host and doesn't require SNAT. This parameter uses the iptables interface matching syntax, which allows `+` as a wildcard. Most users will not need to set this. Example: if your data interfaces are eth0 and eth1 and you want to exclude the docker bridge, you could set this to `eth+` | string | string | `""` | | iptablesPostWriteCheckInterval | Period after Felix has done a write to the data plane that it schedules an extra read back to check the write was not clobbered by another process. This should only occur if another application on the system doesn't respect the iptables lock. | `5s`, `10s`, `1m` etc. | duration | `1s` | | iptablesRefreshInterval | Period at which Felix re-checks all iptables state to ensure that no other process has accidentally broken Calico Cloud's rules. Set to 0 to disable iptables refresh. | `5s`, `10s`, `1m` etc. | duration | `90s` | | ipv6Support | IPv6 support for Felix | `true`, `false` | boolean | `true` | | logFilePath | The full path to the Felix log. Set to `none` to disable file logging. | string | string | `/var/log/calico/felix.log` | | logPrefix | The log prefix that Felix uses when rendering LOG rules. | string | string | `calico-packet` | | logSeverityFile | The log severity above which logs are sent to the log file. | Same as logSeveritySys | string | `Info` | | logSeverityScreen | The log severity above which logs are sent to the stdout. | Same as logSeveritySys | string | `Info` | | logSeveritySys | The log severity above which logs are sent to the syslog. Set to `none` for no logging to syslog. | Debug, Info, Warning, Error, Fatal | string | `Info` | | logDebugFilenameRegex | controls which source code files have their Debug log output included in the logs. Only logs from files with names that match the given regular expression are included. The filter only applies to Debug level logs. | regex | string | `""` | | maxIpsetSize | Maximum size for the ipsets used by Felix. Should be set to a number that is greater than the maximum number of IP addresses that are ever expected in a selector. | int | int | `1048576` | | metadataAddr | The IP address or domain name of the server that can answer VM queries for cloud-init metadata. In OpenStack, this corresponds to the machine running nova-api (or in Ubuntu, nova-api-metadata). A value of `none` (case insensitive) means that Felix should not set up any NAT rule for the metadata path. | IPv4, hostname, none | string | `127.0.0.1` | | metadataPort | The port of the metadata server. This, combined with global.MetadataAddr (if not 'None'), is used to set up a NAT rule, from 169.254.169.254:80 to MetadataAddr:MetadataPort. In most cases this should not need to be changed. | int | int | `8775` | | natOutgoingAddress | The source address to use for outgoing NAT. By default an iptables MASQUERADE rule determines the source address which will use the address on the host interface the traffic leaves on. | IPV4 | string | `""` | | policySyncPathPrefix | File system path where Felix notifies services of policy changes over Unix domain sockets. This is required only if you're configuring [L7 logs](https://docs.tigera.io/calico-cloud/observability/elastic/l7/configure), or [egress gateways](https://docs.tigera.io/calico-cloud/networking/egress/). Set to `""` to disable. | string | string | `""` | | prometheusGoMetricsEnabled | Set to `false` to disable Go runtime metrics collection, which the Prometheus client does by default. This reduces the number of metrics reported, reducing Prometheus load. | boolean | boolean | `true` | | prometheusMetricsEnabled | Set to `true` to enable the experimental Prometheus metrics server in Felix. | boolean | boolean | `false` | | prometheusMetricsHost | TCP network address that the Prometheus metrics server should bind to. | IPv4, IPv6, Hostname | string | `""` | | prometheusMetricsPort | TCP port that the Prometheus metrics server should bind to. | int | int | `9091` | | prometheusProcessMetricsEnabled | Set to `false` to disable process metrics collection, which the Prometheus client does by default. This reduces the number of metrics reported, reducing Prometheus load. | boolean | boolean | `true` | | prometheusReporterEnabled | Set to `true` to enable configure Felix to keep count of recently denied packets and publish these as Prometheus metrics. Note that denied packet metrics are independent of the `dropActionOverride` setting. Specifically, if packets that would normally be denied are being allowed through by a setting of `Accept` or `LogAndAccept`, those packets still get counted as denied packets. | `true`, `false` | boolean | `false` | | prometheusReporterPort | The TCP port on which to report denied packet metrics, if `prometheusReporterEnabled` is set to `true`. | | | `9092` | | removeExternalRoutes | Whether or not to remove device routes that have not been programmed by Felix. Disabling this will allow external applications to also add device routes. | bool | boolean | `true` | | reportingInterval | Interval at which Felix reports its status into the datastore. 0 means disabled and is correct for Kubernetes-only clusters. Must be non-zero in OpenStack deployments. | `5s`, `10s`, `1m` etc. | duration | `30s` | | reportingTTL | Time-to-live setting for process-wide status reports. | `5s`, `10s`, `1m` etc. | duration | `90s` | | routeRefreshInterval | Period at which Felix re-checks the routes in the data plane to ensure that no other process has accidentally broken Calico Cloud's rules. Set to 0 to disable route refresh. | `5s`, `10s`, `1m` etc. | duration | `90s` | | ipsecMode | Controls which mode IPsec is operating on. The only supported value is `PSK`. An empty value means IPsec is not enabled. | PSK | string | `""` | | ipsecAllowUnsecuredTraffic | When set to `false`, only IPsec-protected traffic will be allowed on the packet paths where IPsec is supported. When set to `true`, IPsec will be used but non-IPsec traffic will be accepted. In general, setting this to `true` is less safe since it allows an attacker to inject packets. However, it is useful when transitioning from non-IPsec to IPsec since it allows traffic to flow while the cluster negotiates the IPsec mesh. | `true`, `false` | boolean | `false` | | ipsecIKEAlgorithm | IPsec IKE algorithm. Default is NIST suite B recommendation. | string | string | `aes128gcm16-prfsha256-ecp256` | | ipsecESPAlgorithm | IPsec ESP algorithm. Default is NIST suite B recommendation. | string | string | `aes128gcm16-ecp256` | | ipsecLogLevel | Controls log level for IPsec components. Set to `None` for no logging. | `None`, `Notice`, `Info`, `Debug`, `Verbose` | string | `Info` | | ipsecPSKFile | The path to the pre shared key file for IPsec. | string | string | `""` | | flowLogsFileEnabled | Set to `true`, enables flow logs. If set to `false` no flow logging will occur. Flow logs are written to a file `flows.log` and sent to Elasticsearch. The location of this file can be configured using the `flowLogsFileDirectory` field. File rotation settings for this `flows.log` file can be configured using the fields `flowLogsFileMaxFiles` and `flowLogsFileMaxFileSizeMB`. Note that flow log exports to Elasticsearch are dependent on flow logs getting written to this file. Setting this parameter to `false` will disable flow logs. | `true`, `false` | boolean | `false` | | flowLogsFileDirectory | Set the directory where flow logs files are stored on Linux nodes. This parameter only takes effect when `flowLogsFileEnabled` is set to `true`. | string | string | `/var/log/calico/flowlogs` | | flowLogsPositionFilePath | Specify the position of the external pipeline that reads flow logs on Linux nodes. This parameter only takes effect when `FlowLogsDynamicAggregationEnabled` is set to `true`. | string | string | `/var/log/calico/flows.log.pos` | | flowLogsFileMaxFiles | Set the number of log files to keep. This parameter only takes effect when `flowLogsFileEnabled` is set to `true`. | int | int | `5` | | flowLogsFileMaxFileSizeMB | Set the max size in MB of flow logs files before rotation. This parameter only takes effect when `flowLogsFileEnabled` is set to `true`. | int | int | `100` | | flowLogsFlushInterval | The period, in seconds, at which Felix exports the flow logs. | int | int | `300s` | | flowLogsFileAggregationKindForAllowed | How much to aggregate the flow logs sent to Elasticsearch for allowed traffic. Bear in mind that changing this value may have a dramatic impact on the volume of flow logs sent to Elasticsearch. | 0-2 | [AggregationKind](#aggregationkind) | 2 | | flowLogsFileAggregationKindForDenied | How much to aggregate the flow logs sent to Elasticsearch for denied traffic. Bear in mind that changing this value may have a dramatic impact on the volume of flow logs sent to Elasticsearch. | 0-2 | [AggregationKind](#aggregationkind) | 1 | | flowLogsFileIncludeService | When set to `true`, include destination service information in the aggregated flow log. Note that service information will only be included when the flow can be explicitly determined to be bound to a service (e.g. pre-DNAT destination matches a service ClusterIP). | `true`, `false` | boolean | `false` | | flowLogsFileIncludeLabels | When set to `true`, include source and destination endpoint labels in the aggregated flow log. Note that only Kubernetes endpoints or network sets are included; arbitrary networks do not contain labels. | `true`, `false` | boolean | `false` | | flowLogsFileIncludePolicies | When set to `true`, include all policies in the aggregated flow logs that acted upon and matches the flow log traffic. | `true`, `false` | boolean | `false` | | flowLogsDestDomainsByClient | When set to true, top-level domains are strictly associated with the source IP that originally queried the domains. (default: true) | | | | | flowLogsEnableNetworkSets | When set to `true`, include an arbitrary network set in the aggregated flow log that matches the IP address of the flow log endpoint. | `true`, `false` | boolean | `false` | | flowLogsCollectProcessInfo | When set to `true`, Felix will load the kprobe BPF programs to collect process info. | `true`, `false` | boolean | `false` | | flowLogsCollectTcpStats | When set to `true`, Felix will collect the TCP socket stats. | `true`, `false` | boolean | `true` | | flowLogsCollectProcessPath | When set to `true`, along with flowLogsCollectProcessInfo, each flow log will include the full path of the executable and the arguments with which the executable was invoked. | `true`, `false` | boolean | `false` | | flowLogsFilePerFlowProcessLimit | Specify the maximum number of flow log entries with distinct process information beyond which process information will be aggregated | int | int | `2` | | flowLogsFileNatOutgoingPortLimit | Specify the maximum number of distinct post SNAT ports that will appear in the flowLogs | int | int | `3` | | flowLogsFilePerFlowProcessArgsLimit | Specify the maximum number of unique arguments in the flowlogs beyond which process arguments will be aggregated | int | int | `5` | | flowLogsFileDomainsLimit | Specify the maximum number of top-level domains to include in a flow log. This only applies to source reported flows to destinations external to the cluster. | int | int | `5` | | flowLogsPolicyEvaluationMode | Defines how policies are evaluated and reflected in flow logs. `OnNewConnection`: In this mode, staged policies are only evaluated when new connections are made in the data plane. Staged/active policy changes will not be reflected in the `pending_policies` field of flow logs for long lived connections. `Continuous`: Felix evaluates active flows on a regular basis to determine the rule traces in the flow logs. Any policy updates that impact a flow will be reflected in the `pending_policies` field, offering a near-real-time view of policy changes across flows. | `Continuous`, `OnNewConnection` | string | `Continuous` | | statsDumpFilePath | Specify the position of the file used for dumping flow log statistics on Linux nodes. Note this is an internal setting that users shouldn't need to modify. | string | string | `/var/log/calico/stats/dump` | | routeTableRange | *deprecated in favor of `RouteTableRanges`* Calico programs additional Linux route tables for various purposes. `RouteTableRange` specifies the indices of the route tables that Calico should use. | | [RouteTableRanges](#routetablerange) | `""` | | routeTableRanges | Calico programs additional Linux route tables for various purposes. `RouteTableRanges` specifies a set of table index ranges that Calico should use. Deprecates `RouteTableRange`, overrides `RouteTableRange` | | [RouteTableRanges](#routetableranges) | `[{"min": 1, "max": 250}]` | | routeSyncDisabled | Set to `true` to disable Calico programming routes to local workloads. | boolean | boolean | `false` | | serviceLoopPrevention | When [service IP advertisement is enabled](https://docs.tigera.io/calico-cloud/networking/configuring/advertise-service-ips), prevent routing loops to service IPs that are not in use, by dropping or rejecting packets that do not get DNAT'd by kube-proxy. Unless set to "Disabled", in which case such routing loops continue to be allowed. | `Drop`, `Reject`, `Disabled` | string | `Drop` | | workloadSourceSpoofing | Controls whether pods can enable source IP address spoofing with the `cni.projectcalico.org/allowedSourcePrefixes` annotation. When set to `Any`, pods can use this annotation to send packets from any IP address. | `Any`, `Disabled` | string | `Disabled` | | vxlanEnabled | Optional, you shouldn't need to change this setting as Felix calculates if VXLAN should be enabled based on the existing IP Pools. When set, this overrides whether Felix should create the VXLAN tunnel device for VXLAN networking. | `true`, `false`, unset | optional boolean | unset | | vxlanMTU | MTU to use for the IPv4 VXLAN tunnel device. Zero value means auto-detect. Also controls NodePort MTU when eBPF enabled. | int | int | `0` | | vxlanMTUV6 | MTU to use for the IPv6 VXLAN tunnel device. Zero value means auto-detect. Also controls NodePort MTU when eBPF enabled. | int | int | `0` | | vxlanPort | Port to use for VXLAN traffic. A value of `0` means "use the kernel default". | int | int | `4789` | | vxlanVNI | Virtual network ID to use for VXLAN traffic. A value of `0` means "use the kernel default". | int | int | `4096` | | allowVXLANPacketsFromWorkloads | Set to `true` to allow VXLAN encapsulated traffic from workloads. | boolean | boolean | `false` | | allowIPIPPacketsFromWorkloads | Set to `true` to allow IPIP encapsulated traffic from workloads. | boolean | boolean | `false` | | windowsFlowLogsFileDirectory | Set the directory where flow logs files are stored on Windows nodes. This parameter only takes effect when `flowLogsFileEnabled` is set to `true`. | string | string | `c:\\TigeraCalico\\flowlogs` | | windowsFlowLogsPositionFilePath | Specify the position of the external pipeline that reads flow logs on Windows nodes. This parameter only takes effect when `FlowLogsDynamicAggregationEnabled` is set to `true`. | string | string | `c:\\TigeraCalico\\flowlogs\\flows.log.pos` | | windowsStatsDumpFilePath | Specify the position of the file used for dumping flow log statistics on Windows nodes. Note this is an internal setting that users shouldn't need to modify. | string | string | `c:\\TigeraCalico\\stats\\dump` | | windowsDNSCacheFile | Specify the name of the file that Calico uses to preserve learnt DNS information when restarting. | string | string | `c:\\TigeraCalico\\felix-dns-cache.txt` | | windowsDNSExtraTTL | Specify extra time in seconds to keep IPs and alias names that are learnt from DNS, in addition to each name or IP's advertised TTL. | int | int | `120` | | wireguardEnabled | Enable encryption on WireGuard supported nodes in cluster. When enabled, pod to pod traffic will be sent over encrypted tunnels between the nodes. | `true`, `false` | boolean | `false` | | wireguardEnabledV6 | Enable encryption for IPv6 on WireGuard supported nodes in cluster. When enabled, pod to pod traffic will be sent over encrypted tunnels between the nodes. | `true`, `false` | boolean | `false` | | wireguardInterfaceName | Name of the WireGuard interface created by Felix. If you change the name, and want to clean up the previously-configured interface names on each node, this is a manual process. Felix expect the name to end with either .cali or .calico suffix. | string | string | wireguard.cali | | wireguardInterfaceNameV6 | Name of the IPv6 WireGuard interface created by Felix. If you change the name, and want to clean up the previously-configured interface names on each node, this is a manual process. Felix expect the name to end with either .cali or .calico suffix. | string | string | wg-v6.cali | | wireguardListeningPort | Port used by WireGuard tunnels. Felix sets up WireGuard tunnel on each node specified by this port. Available for configuration only in the global FelixConfiguration resource; setting it per host, config-file or environment variable will not work. | 1-65535 | int | 51820 | | wireguardListeningPortV6 | Port used by IPv6 WireGuard tunnels. Felix sets up an IPv6 WireGuard tunnel on each node specified by this port. Available for configuration only in the global FelixConfiguration resource; setting it per host, config-file or environment variable will not work. | 1-65535 | int | 51821 | | wireguardMTU | MTU set on the WireGuard interface created by Felix. Zero value means auto-detect. See [Configuring MTU](https://docs.tigera.io/calico-cloud/networking/configuring/mtu). | int | int | 0 | | wireguardMTUV6 | MTU set on the IPv6 WireGuard interface created by Felix. Zero value means auto-detect. See [Configuring MTU](https://docs.tigera.io/calico-cloud/networking/configuring/mtu). | int | int | 0 | | wireguardRoutingRulePriority | WireGuard routing rule priority value set up by Felix. If you change the default value, set it to a value most appropriate to routing rules for your nodes. | 1-32765 | int | 99 | | wireguardHostEncryptionEnabled | **Experimental**: Adds host-namespace workload IP's to WireGuard's list of peers. Should **not** be enabled when WireGuard is enabled on a cluster's control plane node, as networking deadlock can occur. | true, false | boolean | false | | wireguardKeepAlive | WireguardKeepAlive controls Wireguard PersistentKeepalive option. Set 0 to disable. \[Default: 0] | `5s`, `10s`, `1m` etc. | duration | `0` | | xdpRefreshInterval | Period at which Felix re-checks the XDP state in the data plane to ensure that no other process has accidentally broken Calico Cloud's rules. Set to 0 to disable XDP refresh. | `5s`, `10s`, `1m` etc. | duration | `90s` | | xdpEnabled | When `bpfEnabled` is `false`: enable XDP acceleration for host endpoint policies. When `bpfEnabled` is `true`, XDP is automatically used for Calico policy where that makes sense, regardless of this setting. \[Default: `true`] | true,false | boolean | `true` | | dnsCacheFile | The name of the file that Felix uses to preserve learnt DNS information when restarting. | file name | string | `/var/run/calico/felix-dns-cache.txt` | | dnsCacheSaveInterval | The period, in seconds, at which Felix saves learnt DNS information to the cache file. | `5s`, `10s`, `1m` etc. | duration | `60s` | | dnsCacheEpoch | An arbitrary number that can be changed, at runtime, to tell Felix to discard all its learnt DNS information. | int | int | `0` | | dnsExtraTTL | Extra time to keep IPs and alias names that are learnt from DNS, in addition to each name or IP's advertised TTL. | `5s`, `10s`, `1m` etc. | duration | `0s` | | dnsTrustedServers | The DNS servers that Felix should trust. Each entry here must be `[:]` - indicating an explicit DNS server IP - or `k8s-service:[/][:port]` - indicating a Kubernetes DNS service. `` defaults to the first service port, or 53 for an IP, and `` to `kube-system`. An IPv6 address with a port must use the square brackets convention, for example `[fd00:83a6::12]:5353`. Note that Felix (calico-node) will need RBAC permission to read the details of each service specified by a `k8s-service:...` form. | IPs or service names | comma-separated strings | `k8s-service:kube-dns` | | dnsLogsFileEnabled | Set to `true`, enables DNS logs. If set to `false` no DNS logging will occur. DNS logs are written to a file `dns.log` and sent to Elasticsearch. The location of this file can be configured using the `DNSLogsFileDirectory` field. File rotation settings for this `dns.log` file can be configured using the fields `DNSLogsFileMaxFiles` and `DNSLogsFileMaxFileSizeMB`. Note that DNS log exports to Elasticsearch are dependent on DNS logs getting written to this file. Setting this parameter to `false` will disable DNS logs. | `true`, `false` | boolean | `false` | | dnsLogsFileDirectory | The directory where DNS logs files are stored. This parameter only takes effect when `DNSLogsFileEnabled` is `true`. | directory | string | `/var/log/calico/dnslogs` | | dnsLogsFileMaxFiles | The number of files to keep when rotating DNS log files. This parameter only takes effect when `DNSLogsFileEnabled` is `true`. | int | int | `5` | | dnsLogsFileMaxFileSizeMB | The max size in MB of DNS log files before rotation. This parameter only takes effect when `DNSLogsFileEnabled` is `true`. | int | int | `100` | | dnsLogsFlushInterval | The period, in seconds, at which Felix exports DNS logs. | int | int | `300s` | | dnsLogsFileAggregationKind | How much to aggregate DNS logs. Bear in mind that changing this value may have a dramatic impact on the volume of flow logs sent to Elasticsearch. `0` means no aggregation, `1` means aggregate similar DNS logs from workloads in the same ReplicaSet. | `0`,`1` | int | `1` | | dnsLogsFileIncludeLabels | Whether to include client and server workload labels in DNS logs. | `true`, `false` | boolean | `true` | | dnsLogsFilePerNodeLimit | Limit on the number of DNS logs that can be emitted within each flush interval. When this limit has been reached, Felix counts the number of unloggable DNS responses within the flush interval, and emits a WARNING log with that count at the same time as it flushes the buffered DNS logs. | int | int | `0` (no limit) | | dnsLogsLatency | Indicates to include measurements of DNS request/response latency in each DNS log. | `true`, `false` | boolean | `true` | | dnsPolicyMode | DNSPolicyMode specifies how DNS policy programming will be handled. | `Inline`, `NoDelay`, `DelayDNSResponse`, `DelayDeniedPacket` | [DNSPolicyMode](#dnspolicymode) | `Inline` | | bpfDNSPolicyMode | BPFDNSPolicyMode specifies how DNS policy programming will be handled. | `NoDelay`, `Inline` | [BPFDNSPolicyMode](#bpfdnspolicymode) | `Inline` | | nftablesDNSPolicyMode | Specifies how DNS policy programming will be handled for NFTables. Options include `NoDelay`, `DelayDNSResponse`, and `DelayDeniedPacket`. See [NFTablesDNSPolicyMode](#nftablesdnspolicymode) for details. | `NoDelay`, `DelayDNSResponse`, `DelayDeniedPacket` | [NFTablesDNSPolicyMode](#nftablesdnspolicymode) | `DelayDeniedPacket` | | dnsPolicyNfqueueID | DNSPolicyNfqueueID is the NFQUEUE ID to use for DNS Policy re-evaluation when the domains IP hasn't been programmed to ipsets yet. This value can be changed to avoid conflicts with other users of NFQUEUEs. Used when `DNSPolicyMode` is `DelayDeniedPacket`. | 0-65535 | int | `100` | | dnsPolicyNfqueueSize | DNSPolicyNfqueueID is the size of the NFQUEUE for DNS policy re-evaluation. This is the maximum number of denied packets that may be queued up pending re-evaluation. Used when `DNSPolicyMode` is `DelayDeniedPacket`. | 0-65535 | int | `100` | | dnsPacketsNfqueueID | DNSPacketsNfqueueID is the NFQUEUE ID to use for capturing DNS packets to ensure programming IPSets occurs before the response is released. Used when `DNSPolicyMode` is `DelayDNSResponse`. | 0-65535 | int | `101` | | dnsPacketsNfqueueSize | DNSPacketsNfqueueSize is the size of the NFQUEUE for captured DNS packets. This is the maximum number of DNS packets that may be queued awaiting programming in the data plane. Used when `DNSPolicyMode` is `DelayDNSResponse`. | 0-65535 | int | `100` | | dnsPacketsNfqueueMaxHoldDuration | DNSPacketsNfqueueMaxHoldDuration is the max length of time to hold on to a DNS response while waiting for the data plane to be programmed. Used when `DNSPolicyMode` is `DelayDNSResponse`. | `5s`, `10s`, `1m` etc. | duration | `3s` | | bpfEnabled | Enable eBPF data plane mode. eBPF mode has some limitations, see the [HOWTO guide](https://docs.tigera.io/calico-cloud/operations/ebpf/enabling-ebpf) for more details. | true, false | boolean | false | | bpfDisableUnprivileged | If true, Felix sets the kernel.unprivileged\_bpf\_disabled sysctl to disable unprivileged use of BPF. This ensures that unprivileged users cannot access Calico's BPF maps and cannot insert their own BPF programs to interfere with the ones that Calico Cloud installs. | true, false | boolean | true | | bpfLogLevel | In eBPF data plane mode, the log level used by the BPF programs. The logs are emitted to the BPF trace pipe, accessible with the command `tc exec bpf debug`. This is a tech preview feature and subject to change in future releases. | Off,Info,Debug | string | Off | | bpfDataIfacePattern | In eBPF data plane mode, controls which interfaces Felix should attach BPF programs to catch traffic to/from the external network. This needs to match the interfaces that Calico workload traffic flows over as well as any interfaces that handle incoming traffic to NodePorts and services from outside the cluster. It should not match the workload interfaces (usually named cali...).. This is a tech preview feature and subject to change in future releases. | regular expression | string | ^(en.\*\|eth.\*\|tunl0$) | | bpfConnectTimeLoadBalancingEnabled | In eBPF data plane mode, controls whether Felix installs the connect-time load balancer. In the current release, the connect-time load balancer is required for the host to reach kubernetes services. This is a tech preview feature and subject to change in future releases. | true,false | boolean | true | | bpfExternalServiceMode | In eBPF data plane mode, controls how traffic from outside the cluster to NodePorts and ClusterIPs is handled. In Tunnel mode, packet is tunneled from the ingress host to the host with the backing pod and back again. In DSR mode, traffic is tunneled to the host with the backing pod and then returned directly; this requires a network that allows direct return. | Tunnel,DSR | string | Tunnel | | bpfKubeProxyIptablesCleanupEnabled | In eBPF data plane mode, controls whether Felix will clean up the iptables rules created by the Kubernetes `kube-proxy`; should only be enabled if `kube-proxy` is not running. This is a tech preview feature and subject to change in future releases. | true,false | boolean | true | | bpfKubeProxyMinSyncPeriod | In eBPF data plane mode, controls the minimum time between data plane updates for Felix's embedded `kube-proxy` implementation. | `5s`, `10s`, `1m` etc. | duration | `1s` | | BPFKubeProxyEndpointSlicesEnabled | In eBPF data plane mode, controls whether Felix's embedded kube-proxy derives its services from Kubernetes' EndpointSlices resources. Using EndpointSlices is more efficient but it requires EndpointSlices support to be enabled at the Kubernetes API server. | true,false | boolean | false | | bpfMapSizeConntrack | In eBPF data plane mode, controls the size of the conntrack map. | int | int | 512000 | | bpfMapSizeIPSets | In eBPF data plane mode, controls the size of the ipsets map. | int | int | 1048576 | | bpfMapSizeNATAffinity | In eBPF data plane mode, controls the size of the NAT affinity map. | int | int | 65536 | | bpfMapSizeNATFrontend | In eBPF data plane mode, controls the size of the NAT front end map. | int | int | 65536 | | bpfMapSizeNATBackend | In eBPF data plane mode, controls the size of the NAT back end map. | int | int | 262144 | | bpfMapSizeRoute | In eBPF data plane mode, controls the size of the route map. | int | int | 262144 | | bpfPolicyDebugEnabled | In eBPF data plane mode, controls whether felix will collect policy dump for each interface. | true, false | boolean | true | | routeSource | Where Felix gets is routing information from for VXLAN and the BPF data plane. The CalicoIPAM setting is more efficient because it supports route aggregation, but it only works when Calico's IPAM or host-local IPAM is in use. Use the WorkloadIPs setting if you are using Calico's VXLAN or BPF data plane and not using Calico IPAM or host-local IPAM. | CalicoIPAM,WorkloadIPs | string | `CalicoIPAM` | | mtuIfacePattern | Pattern used to discover the host's interface for MTU auto-detection. | regex | string | ^((en\|wl\|ww\|sl\|ib)\[opsvx].\*\|(eth\|wlan\|wwan).\*) | | bpfForceTrackPacketsFromIfaces | Forces traffic from these interfaces in BPF mode to skip Calico's iptables NOTRACK rule, allowing traffic from those interfaces to be tracked by Linux conntrack. Use only for interfaces that are not used for the Calico fabric, for example, a docker bridge device for non-Calico-networked containers. | A list of strings | A list of strings | docker+ | | bpfDisableGROForIfaces | BPFDisableGROForIfaces is a regular expression that controls which interfaces Felix should disable the Generic Receive Offload \[GRO] option. It should not match the workload interfaces (usually named cali...). | regex | string | "" | | egressIPSupport | Defines three different support modes for egress gateway function. `Disabled` means egress gateways are not supported. `EnabledPerNamespace` means egress gateway function is enabled and can be configured on a per-namespace basis (but per-pod egress annotations are ignored). `EnabledPerNamespaceOrPerPod` means egress gateway function is enabled and can be configured per-namespace or per-pod (with per-pod egress annotations overriding namespace annotations). | Disabled, EnabledPerNamespace, EnabledPerNamespaceOrPerPod | string | `Disabled` | | egressIPVXLANPort | Port to use for egress gateway VXLAN traffic. A value of `0` means "use the kernel default". | int | int | `4790` | | egressIPVXLANVNI | Virtual network ID to use for egress gateway VXLAN traffic. A value of `0` means "use the kernel default". | int | int | `4097` | | egressIPRoutingRulePriority | Controls the priority value to use for the egress gateway routing rule. | int | int | `100` | | egressGatewayPollInterval | Controls the interval at which Felix will poll remote egress gateways to check their health. Only Egress Gateways with a named "health" port will be polled in this way. Egress Gateways that fail the health check will be taken our of use as if they have been deleted. | `5s`, `10s`, `1m` etc. | duration | `10s` | | egressGatewayPollFailureCount | Controls the minimum number of poll failures before a remote Egress Gateway is considered to have failed. | int | int | `3` | | captureDir | Controls the directory where packet capture files are stored. | string | string | `/var/log/calico/pcap` | | captureMaxSizeBytes | Controls the maximum size in bytes for a packet capture file before rotation. | int | int | `10000000` | | captureRotationSeconds | Controls the rotation period in seconds for a packet capture file. | int | int | `3600` | | captureMaxFiles | Controls the maximum number rotated packet capture files. | int | int | `2` | \* When `dropActionOverride` is set to `LogAndDrop` or `LogAndAccept`, the `syslog` entries look something like the following. ```text May 18 18:42:44 ubuntu kernel: [ 1156.246182] calico-drop: IN=tunl0 OUT=cali76be879f658 MAC= SRC=192.168.128.30 DST=192.168.157.26 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=56743 DF PROTO=TCP SPT=56248 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0 MARK=0xa000000 ``` \*\* Duration is denoted by the numerical amount followed by the unit of time. Valid units of time include nanoseconds (ns), microseconds (µs), milliseconds (ms), seconds (s), minutes (m), and hours (h). Units of time can also be used together e.g. `3m30s` to represent 3 minutes and 30 seconds. Any amounts of time that can be converted into larger units of time will be converted e.g. `90s` will become `1m30s`. `genericXDPEnabled` and `xdpRefreshInterval` are only relevant when `bpfEnabled` is `false` and `xdpEnabled` is `true`; in other words when XDP is being used to accelerate denial-of-service prevention policies in the iptables data plane. When `bpfEnabled` is `true` the "xdp" settings all have no effect; in BPF mode the implementation of policy is always accelerated, using the best available BPF technology. ### Health Timeout Overrides[​](#health-timeout-overrides) Felix has internal liveness and readiness watchdog timers that monitor its various loops. If a loop fails to "check in" within the allotted timeout then Felix will report non-Ready or non-Live on its health port (which is monitored by Kubelet in a Kubernetes system). If Felix reports non-Live, this can result in the Pod being restarted. In Kubernetes, if you see the calico-node Pod readiness or liveness checks fail intermittently, check the calico-node Pod log for a log from Felix that gives the overall health status (the list of components will depend on which features are enabled): ```text +---------------------------+---------+----------------+-----------------+--------+ | COMPONENT | TIMEOUT | LIVENESS | READINESS | DETAIL | +---------------------------+---------+----------------+-----------------+--------+ | CalculationGraph | 30s | reporting live | reporting ready | | | FelixStartup | 0s | reporting live | reporting ready | | | InternalDataplaneMainLoop | 1m30s | reporting live | reporting ready | | +---------------------------+---------+----------------+-----------------+--------+ ``` If some health timeouts show as "timed out" it may help to apply an override using the `healthTimeoutOverrides` field: ```yaml ... spec: healthTimeoutOverrides: - name: InternalDataplaneMainLoop timeout: "5m" - name: CalculationGraph timeout: "1m30s" ... ``` A timeout value of 0 disables the timeout. ### ProtoPort[​](#protoport) | Field | Description | Accepted Values | Schema | | -------- | -------------------- | ------------------------------------ | ------ | | port | The exact port match | 0-65535 | int | | protocol | The protocol match | tcp, udp, sctp | string | | net | The CIDR match | any valid CIDR (e.g. 192.168.0.0/16) | string | Keep in mind that in the following example, `net: ""` and `net: "0.0.0.0/0"` are processed as the same in the policy enforcement. ```yaml ... spec: failsafeInboundHostPorts: - net: "192.168.1.1/32" port: 22 protocol: tcp - net: "" port: 67 protocol: udp failsafeOutboundHostPorts: - net: "0.0.0.0/0" port: 67 protocol: udp ... ``` ### AggregationKind[​](#aggregationkind) | Value | Description | | ----- | ---------------------------------------------------------------------------------------- | | 0 | No aggregation | | 1 | Aggregate all flows that share a source port on each node | | 2 | Aggregate all flows that share source ports or are from the same ReplicaSet on each node | ### DNSPolicyMode[​](#dnspolicymode) | Value | Description | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Inline (default) | Felix parses DNS response inline with DNS response packet processing within IPTables. This guarantees the DNS rules reflect any change immediately. This mode works for iptables only and matches the same mode for `BPFDNSPolicyMode`. | | DelayDeniedPacket | Felix delays any denied packet that traversed a policy that included egress domain matches, but did not match. The packet is released after a fixed time, or after the destination IP address was programmed. | | DelayDNSResponse | Felix delays any DNS response until related IPSets are programmed. This introduces some latency to all DNS packets (even when no IPSet programming is required), but it ensures policy hit statistics are accurate. This is the recommended setting when you are making use of staged policies or policy rule hit statistics. | | NoDelay | Felix does not introduce any delay to the packets. DNS rules may not have been programmed by the time the first packet traverses the policy rules. Client applications need to handle reconnection attempts if initial connection attempts fail. This may be problematic for some applications or for very low DNS TTLs. | On Windows, or when using the eBPF data plane, this setting is ignored and `NoDelay` is always used. This field has no effect in NFTables mode. Please use `NFTablesDNSPolicyMode` instead. A linux kernel version of 3.13 or greater is required to use `DelayDNSResponse`. For earlier kernel versions, this value is modified to `DelayDeniedPacket`. ### NFTablesDNSPolicyMode[​](#nftablesdnspolicymode) | Value | Description | | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | DelayDeniedPacket (default) | Felix delays any denied packet that traversed a policy that included egress domain matches, but did not match. The packet is released after a fixed time, or after the destination IP address was programmed. | | DelayDNSResponse | Felix delays any DNS response until related IPSets are programmed. This introduces some latency to all DNS packets (even when no IPSet programming is required), but it ensures policy hit statistics are accurate. This is the recommended setting when you are making use of staged policies or policy rule hit statistics. | | NoDelay | Felix does not introduce any delay to the packets. DNS rules may not have been programmed by the time the first packet traverses the policy rules. Client applications need to handle reconnection attempts if initial connection attempts fail. This may be problematic for some applications or for very low DNS TTLs. | ### RouteTableRange[​](#routetablerange) The `RouteTableRange` option is now deprecated in favor of [RouteTableRanges](#routetableranges). | Field | Description | Accepted Values | Schema | | ----- | -------------------- | --------------- | ------ | | min | Minimum index to use | 1-250 | int | | max | Maximum index to use | 1-250 | int | ### RouteTableRanges[​](#routetableranges) `RouteTableRanges` is a list of `RouteTableRange` objects: | Field | Description | Accepted Values | Schema | | ----- | -------------------- | --------------- | ------ | | min | Minimum index to use | 1 - 4294967295 | int | | max | Maximum index to use | 1 - 4294967295 | int | Each item in the `RouteTableRanges` list designates a range of routing tables available to Calico. By default, Calico will use a single range of `1-250`. If a range spans Linux's reserved table range (`253-255`) then those tables are automatically excluded from the list. It's possible that other table ranges may also be reserved by third-party systems unknown to Calico. In that case, multiple ranges can be defined to target tables below and above the sensitive ranges: ```sh target tables 65-99, and 256-1000, skipping 100-255 calicoctl patch felixconfig default --type=merge -p '{"spec":{"routeTableRanges": [{"min": 65, "max": 99}, {"min": 256, "max": 1000}] }} ``` *Note*, for performance reasons, the maximum total number of routing tables that Felix will accept is 65535 (or 2\*16). Specifying both the `RouteTableRange` and `RouteTableRanges` arguments is not supported and will result in an error from the api. ### AWS IAM Role/Policy for source-destination-check configuration[​](#aws-iam-rolepolicy-for-source-destination-check-configuration) Setting `awsSrcDstCheck` to `Disable` will automatically disable source-destination-check on EC2 instances in a cluster, provided necessary IAM roles and policies are set. One of the policies assigned to IAM role of cluster nodes must contain a statement similar to the following: ```text { "Effect": "Allow", "Action": [ "ec2:DescribeInstances", "ec2:ModifyNetworkInterfaceAttribute" ], "Resource": "*" } ``` If there are no policies attached to node roles containing the above statement, attach a new policy. For example, if a node role is `test-cluster-nodeinstance-role`, click on the IAM role in AWS console. In the `Permission policies` list, add a new inline policy with the above statement to the new policy JSON definition. For detailed information, see [AWS documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html?icmpid=docs_iam_console). For an EKS cluster, the necessary IAM role and policy is available by default. No further actions are needed. ## Supported operations[​](#supported-operations) | Datastore type | Create | Delete | Delete (Global `default`) | Update | Get/List | Notes | | --------------------- | ------ | ------ | ------------------------- | ------ | -------- | ----- | | Kubernetes API server | Yes | Yes | No | Yes | Yes | | ### Egress gateway policy An EgressGatewayPolicy resource (`EgressGatewayPolicy`) represents a way to select different egress gateways or skip one for different destinations. Rules in an Egress EgressGatewayPolicy are checked in Longest Prefix Match(LPM) fashion like routers. As such it is not valid to use the exact destination in two rules. In order for an EgressGatewayPolicy to be used, its `name` must be added to a pod or namespace by using `egress.projectcalico.org/egressGatewayPolicy` annotation. ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: EgressGatewayPolicy metadata: name: my-egwpolicy spec: rules: - destination: cidr: 10.0.0.0/8 description: "Local: no gateway" - destination: cidr: 11.0.0.0/8 description: "Gateway to on prem" gateway: namespaceSelector: "projectcalico.org/name == 'default'" selector: "egress-code == 'blue'" maxNextHops: 2 - description: "Gateway to internet" gateway: namespaceSelector: "projectcalico.org/name == 'default'" selector: "egress-code == 'red'" gatewayPreference: PreferNodeLocal ``` ## Egress gateway policy definition[​](#egress-gateway-policy-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | | ----- | ------------------------------------------------------------------ | --------------------------------------------------- | ------ | | name | Unique name to describe this resource instance. Must be specified. | Alphanumeric string with optional `.`, `_`, or `-`. | string | ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | ----- | ------------------------------- | --------------- | --------------------------------------------------------- | ------- | | rules | List of egress gateway policies | | [Egress Gateway Policy Rule](#egress-gateway-policy-rule) | | ### Egress gateway policy rule[​](#egress-gateway-policy-rule) | Field | Description | Accepted Values | Schema | Default | | ----------------- | ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- | ------- | | description | A description of rule | | string | | | destination | CIDR representing a destination | | [destination](#destination) | | | gateway | egress gateway to be used for a destination | | [gateway](#egress-gateway) | | | gatewayPreference | Hints about egress gateway selection | `None` for using all available egress gateway replicas from the selected deployment, or `PreferNodeLocal` to use only egress gateway replicas on the same local node as the client pod or namespace if available, otherwise fall back to the default behaviour. | | 'None' | ### Destination[​](#destination) | Field | Description | Accepted Values | Schema | Default | | ----- | --------------------------- | --------------- | ------ | ------- | | cidr | CIDR of destination network | | string | | ### Egress gateway[​](#egress-gateway) | Field | Description | Accepted Values | Schema | Default | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------- | ------ | ------- | | selector | selector to choose an egress gateway deployment | | string | | | namespaceSelector | name space of egress gateway deployment | | string | | | maxNextHops | Specifies the maximum number of egress gateway replicas from the selected deployment that a pod should depend on. Replicas will be chosen in a manner that attempts to balance load across the whole egress gateway replicaset. If unset, or set to "0", egress traffic will behave in the default manner (load balanced over all available gateways). | | string | | ## Supported operations[​](#supported-operations) | Datastore type | Create/Delete | Update | Get/List | Notes | | --------------------- | ------------- | ------ | -------- | ----- | | Kubernetes API server | Yes | Yes | Yes | | ### Global Alert A global alert resource represents a query that is periodically run against data sets collected by Calico Cloud whose findings are added to the Alerts page in the Calico Cloud web console. Alerts may search for the existence of rows in a query, or when aggregated metrics satisfy a condition. Calico Cloud supports alerts on the following data sets: - [Audit logs](https://docs.tigera.io/calico-cloud/observability/elastic/audit-overview) - [DNS logs](https://docs.tigera.io/calico-cloud/observability/elastic/dns/) - [Flow logs](https://docs.tigera.io/calico-cloud/observability/elastic/flow/) - [L7 logs](https://docs.tigera.io/calico-cloud/observability/elastic/l7/) For `kubectl` [commands](https://kubernetes.io/docs/reference/kubectl/overview/), the following case-insensitive aliases can be used to specify the resource type on the CLI: `globalalert.projectcalico.org`, `globalalerts.projectcalico.org` and abbreviations such as `globalalert.p` and `globalalerts.p`. ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: GlobalAlert metadata: name: sample spec: summary: 'Sample' description: 'Sample ${source_namespace}/${source_name_aggr}' severity: 100 dataSet: flows query: action=allow aggregateBy: [source_namespace, source_name_aggr] field: num_flows metric: sum condition: gt threshold: 0 ``` ## GlobalAlert definition[​](#globalalert-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | | ----- | ----------------------- | ----------------------------------------- | ------ | | name | The name of this alert. | Lower-case alphanumeric with optional `-` | string | ### Spec[​](#spec) | Field | Description | Type | Required | Acceptable Values | Default | | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | ----------------------------------------- | ------------------------------------ | -------------------------------- | | type | Type will dictate how the fields of the GlobalAlert will be utilized. Each `type` will have different usages and/or defaults for the other GlobalAlert fields as described in the table. | string | no | RuleBased | RuleBased | | description | Human-readable description of the template. | string | yes | | | | summary | Template for the description field in generated events. See the summary section below for more details. `description` is used if this is omitted. | string | no | | | | severity | Severity of the alert for display in Manager. | int | yes | 1 - 100 | | | dataSet | Which data set to execute the alert against. | string | if `type` is `RuleBased` | audit, dns, flows, l7, vulnerability | | | period | How often the query defined will run, if `type` is `RuleBased`. | duration | no | 1h 2m 3s | 5m, 15m if `type` is `RuleBased` | | lookback | Specifies how far back in time data is to be collected. Must exceed audit log flush interval, `dnsLogsFlushInterval`, or `flowLogsFlushInterval` as appropriate. | duration | no | 1h 2m 3s | 10m | | query | Which data to include from the source data set. Written in a domain-specific query language. See the query section below. | string | no | | | | aggregateBy | An optional list of fields to aggregate results. | string array | no | | | | field | Which field to aggregate results by if using a metric other than count. | string | if metric is one of avg, max, min, or sum | | | | metric | A metric to apply to aggregated results. `count` is the number of log entries matching the aggregation pattern. Others are applied only to numeric fields in the logs. | string | no | avg, max, min, sum, count | | | condition | Compare the value of the metric to the threshold using this condition. | string | if metric defined | eq, not\_eq, lt, lte, gt, gte | | | threshold | A numeric value to compare the value of the metric against. | float | if metric defined | | | | substitutions | An optional list of values to replace variable names in query. | List of [GlobalAlertSubstitution](#globalalertsubstitution) | no | | | ### GlobalAlertSubstitution[​](#globalalertsubstitution) | Field | Description | Type | Required | | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | -------- | | name | The name of the global alert substitution. It will be referenced by the variable names in query. Duplicate names are not allowed in the substitutions list. | string | yes | | values | A list of values for this substitution. Wildcard operators asterisk (`*`) and question mark (`?`) are supported. | string array | yes | ### Status[​](#status) | Field | Description | | --------------- | ------------------------------------------------------------------------------------------- | | lastUpdate | When the alert was last modified on the backend. | | active | Whether the alert is active on the backend. | | healthy | Whether the alert is in an error state or not. | | lastExecuted | When the query for the alert last ran. | | lastEvent | When the condition of the alert was last satisfied and an alert was successfully generated. | | errorConditions | List of errors preventing operation of the updates or search. | ## Query[​](#query) Alerts use a domain-specific query language to select which records from the data set should be used in the alert. This could be used to identify flows with specific features, or to select (or omit) certain namespaces from consideration. The query language is composed of any number of selectors, combined with boolean expressions (`AND`, `OR`, and `NOT`), set expressions (`IN` and `NOTIN`) and bracketed subexpressions. These are translated by Calico Cloud to Elastic DSL queries that are executed on the backend. Set expressions support wildcard operators asterisk (`*`) and question mark (`?`). The asterisk sign matches zero or more characters and the question mark matches a single character. Set values can be embedded into the query string or reference the values in the global alert substitution list. A selector consists of a key, comparator, and value. Keys and values may be identifiers consisting of alphanumerics and underscores (`_`) with the first character being alphabetic or an underscore, or may be quoted strings. Values may also be integer or floating point numbers. Comparators may be `=` (equal), `!=` (not equal), `<` (less than), `<=` (less than or equal), `>` (greater than), or `>=` (greater than or equal). Keys must be indexed fields in their corresponding data set. See the appendix for a list of valid keys in each data set. Examples: - `query: "count > 0"` - `query: "\"servers.ip\" = \"127.0.0.1\""` Selectors may be combined using `AND`, `OR`, and `NOT` boolean expressions, `IN` and `NOTIN` set expressions, and bracketed subexpressions. Examples: - `query: "count > 100 AND client_name=mypod"` - `query: "client_namespace = ns1 OR client_namespace = ns2"` - `query: "count > 100 AND NOT (client_namespace = ns1 OR client_namespace = ns2)"` - `query: "(qtype = A OR qtype = AAAA) AND rcode != NoError"` - `query: "process_name IN {\"proc1?\", \"*proc2\"} AND source_namespace = ns1` - `query: "qname NOTIN ${domains}"` ## Aggregation[​](#aggregation) Results from the query can be aggregated by any number of data fields. Only these data fields will be included in the generated alerts, and each unique combination of aggregations will generate a unique alert. Careful consideration of fields for aggregation will yield the best results. Some good choices for aggregations on the `flows` data set are `[source_namespace, source_name_aggr, source_name]`, `[source_ip]`, `[dest_namespace, dest_name_aggr, dest_name]`, and `[dest_ip]` depending on your use case. For the `dns` data set, `[client_namespace, client_name_aggr, client_name]` is a good choice for an aggregation pattern. ## Metrics and conditions[​](#metrics-and-conditions) Results from the query can be further aggregated using a metric that is applied to a numeric field, or counts the number of rows in an aggregation. Search hits satisfying the condition are output as alerts. | Metric | Description | Applied to Field | | ------ | ---------------------------------- | ---------------- | | count | Counts the number of rows | No | | min | The minimal value of the field | Yes | | max | The maximal value of the field | Yes | | sum | The sum of all values of the field | Yes | | avg | The average value of the field | Yes | | Condition | Description | | --------- | --------------------- | | eq | Equals | | not\_eq | Not equals | | lt | Less than | | lte | Less than or equal | | gt | Greater than | | gte | Greater than or equal | Example: ```yaml apiVersion: projectcalico.org/v3 kind: GlobalAlert metadata: name: frequent-dns-responses spec: description: 'Monitor for NXDomain' summary: 'Observed ${sum} NXDomain responses for ${qname}' severity: 100 dataSet: dns query: rcode = NXDomain AND (rtype = A or rtype = AAAA) aggregateBy: qname field: count metric: sum condition: gte threshold: 100 ``` This alert identifies non-existing DNS responses for Internet addresses that were observed more than 100 times in the past 10 minutes. ### Unconditional alerts[​](#unconditional-alerts) If the `field`, `metric`, `condition`, and `threshold` fields of an alert are left blank then the alert will trigger whenever its query returns any data. Each hit (or aggregation pattern, if `aggregateBy` is non-empty) returned will cause an event to be created. This should be used **only** when the query is highly specific to avoid filling the Alerts page and index with a large number of events. The use of `aggregateBy` is strongly recommended to reduce the number of entries added to the Alerts page. The following example would alert on incoming connections to postgres pods from the Internet that were not denied by policy. It runs hourly to reduce the noise. Noise could be further reduced by removing `source_ip` from the `aggregateBy` clause at the cost of removing `source_ip` from the generated events. ```yaml period: 1h lookback: 75m query: 'dest_labels="application=postgres" AND source_type=net AND action=allow AND proto=tcp AND dest_port=5432' aggregateBy: [dest_namespace, dest_name, source_ip] ``` ## Summary template[​](#summary-template) Alerts may include a summary template to provide context for the alerts in the Calico Cloud web console Alert user interface. Any field in the `aggregateBy` section, or the value of the `metric` may be substituted in the summary using a bracketed variable syntax. Example: ```yaml summary: 'Observed ${sum} NXDomain responses for ${qname}' ``` The `description` field is validated in the same manner. If not provided, the `description` field is used in place of the `summary` field. ## Period and lookback[​](#period-and-lookback) The interval between alerts, and the amount of data considered by the alert may be controlled using the `period` and `lookback` parameters respectively. These fields are formatted as [duration](https://golang.org/pkg/time/#ParseDuration) strings. > A duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as "300ms", "-1.5h" or "2h45m". Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". The default period is 5 minutes, and lookback is 10 minutes. The lookback should always be greater than the sum of the period and the configured `FlowLogsFlushInterval` or `DNSLogsFlushInterval` as appropriate to avoid gaps in coverage. ## Alert records[​](#alert-records) With only aggregations and no metrics, the alert will generate one event per aggregation pattern returned by the query. The record field will contain only the aggregated fields. As before, this should be used with specific queries. The addition of a metric will include the value of that metric in the record, along with any aggregations. This, combined with queries as necessary, will yield the best results in most cases. With no aggregations the alert will generate one event per record returned by the query. The record will be included in its entirety in the record field of the event. This should only be used with very narrow and specific queries. ## Templates[​](#templates) Calico Cloud supports the `GlobalAlertTemplate` resource type. These are used in the Calico Cloud web console to create alerts with prepopulated fields that can be modified to suit your needs. The `GlobalAlertTemplate` resource is configured identically to the `GlobalAlert` resource. Calico Cloud includes some sample Alert templates; add your own templates as needed. ### Sample YAML[​](#sample-yaml-1) **RuleBased GlobalAlert** ```yaml apiVersion: projectcalico.org/v3 kind: GlobalAlertTemplate metadata: name: http.connections spec: description: 'HTTP connections to a target namespace' summary: 'HTTP connections from ${source_namespace}/${source_name_aggr} to /${dest_name_aggr}' severity: 50 dataSet: flows query: dest_namespace="" AND dest_port=80 aggregateBy: [source_namespace, dest_name_aggr, source_name_aggr] field: count metric: sum condition: gte threshold: 1 ``` ## Appendix: Valid fields for queries[​](#appendix-valid-fields-for-queries) ### Audit logs[​](#audit-logs) See [audit.k8s.io group v1](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/apis/audit/v1/types.go) for descriptions of fields. ### DNS logs[​](#dns-logs) See [DNS logs](https://docs.tigera.io/calico-cloud/observability/elastic/dns/dns-logs) for description of fields. ### Flow logs[​](#flow-logs) See [Flow logs](https://docs.tigera.io/calico-cloud/observability/elastic/flow/datatypes) for description of fields. ### L7 logs[​](#l7-logs) See [L7 logs](https://docs.tigera.io/calico-cloud/observability/elastic/l7/datatypes) for description of fields. ### Global network policy A global network policy resource (`GlobalNetworkPolicy`) represents an ordered set of rules which are applied to a collection of endpoints that match a [label selector](#selector). `GlobalNetworkPolicy` is not a namespaced resource. `GlobalNetworkPolicy` applies to [workload endpoint resources](https://docs.tigera.io/calico-cloud/reference/resources/workloadendpoint) in all namespaces, and to [host endpoint resources](https://docs.tigera.io/calico-cloud/reference/resources/hostendpoint). Select a namespace in a `GlobalNetworkPolicy` in the standard selector by using `projectcalico.org/namespace` as the label name and a `namespace` name as the value to compare against, e.g., `projectcalico.org/namespace == "default"`. See [network policy resource](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) for namespaced network policy. `GlobalNetworkPolicy` resources can be used to define network connectivity rules between groups of Calico Cloud endpoints and host endpoints. GlobalNetworkPolicies are organized into [tiers](https://docs.tigera.io/calico-cloud/reference/resources/tier), which provide an additional layer of ordering—in particular, note that the `Pass` action skips to the next [tier](https://docs.tigera.io/calico-cloud/reference/resources/tier), to enable hierarchical security policy. For `kubectl` [commands](https://kubernetes.io/docs/reference/kubectl/overview/), the following case-insensitive aliases may be used to specify the resource type on the CLI: `globalnetworkpolicy.projectcalico.org`, `globalnetworkpolicies.projectcalico.org` and abbreviations such as `globalnetworkpolicy.p` and `globalnetworkpolicies.p`. ## Sample YAML[​](#sample-yaml) This sample policy allows TCP traffic from `frontend` endpoints to port 6379 on `database` endpoints. ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: internal-access.allow-tcp-6379 spec: tier: internal-access selector: role == 'database' types: - Ingress - Egress ingress: - action: Allow metadata: annotations: from: frontend to: database protocol: TCP source: selector: role == 'frontend' destination: ports: - 6379 egress: - action: Allow ``` ## Definition[​](#definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | Default | | ----- | ----------------------------------------- | --------------------------------------------------- | ------ | ------- | | name | The name of the network policy. Required. | Alphanumeric string with optional `.`, `_`, or `-`. | string | | ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------- | --------------------- | --------------------------------------------- | | order | Controls the order of precedence. Calico Cloud applies the policy with the lowest value first. | | float | | | tier | Name of the [tier](https://docs.tigera.io/calico-cloud/reference/resources/tier) this policy belongs to. | | string | `default` | | selector | Selects the endpoints to which this policy applies. | | [selector](#selector) | all() | | serviceAccountSelector | Selects the service account(s) to which this policy applies. Select all service accounts in the cluster with a specific name using the `projectcalico.org/name` label. | | [selector](#selector) | all() | | namespaceSelector | Selects the namespace(s) to which this policy applies. Select a specific namespace by name using the `projectcalico.org/name` label. | | [selector](#selector) | all() | | types | Applies the policy based on the direction of the traffic. To apply the policy to inbound traffic, set to `Ingress`. To apply the policy to outbound traffic, set to `Egress`. To apply the policy to both, set to `Ingress, Egress`. | `Ingress`, `Egress` | List of strings | Depends on presence of ingress/egress rules\* | | ingress | Ordered list of ingress rules applied by policy. | | List of [Rule](#rule) | | | egress | Ordered list of egress rules applied by this policy. | | List of [Rule](#rule) | | | doNotTrack\*\* | Indicates to apply the rules in this policy before any data plane connection tracking, and that packets allowed by these rules should not be tracked. | true, false | boolean | false | | preDNAT\*\* | Indicates to apply the rules in this policy before any DNAT. | true, false | boolean | false | | applyOnForward\*\* | Indicates to apply the rules in this policy on forwarded traffic as well as to locally terminated traffic. | true, false | boolean | false | | performanceHints | Contains a list of hints to Calico's policy engine to help process the policy more efficiently. Hints never change the enforcement behaviour of the policy. The available hints are described [below](#performance-hints). | `AssumeNeededOnEveryNode` | List of strings | | \* If `types` has no value, Calico Cloud defaults as follows. > | Ingress Rules Present | Egress Rules Present | `Types` value | > | --------------------- | -------------------- | ----------------- | > | No | No | `Ingress` | > | Yes | No | `Ingress` | > | No | Yes | `Egress` | > | Yes | Yes | `Ingress, Egress` | \*\* The `doNotTrack` and `preDNAT` and `applyOnForward` fields are meaningful only when applying policy to a [host endpoint](https://docs.tigera.io/calico-cloud/reference/resources/hostendpoint). Only one of `doNotTrack` and `preDNAT` may be set to `true` (in a given policy). If they are both `false`, or when applying the policy to a [workload endpoint](https://docs.tigera.io/calico-cloud/reference/resources/workloadendpoint), the policy is enforced after connection tracking and any DNAT. `applyOnForward` must be set to `true` if either `doNotTrack` or `preDNAT` is `true` because for a given policy, any untracked rules or rules before DNAT will in practice apply to forwarded traffic. ### Rule[​](#rule) A single rule matches a set of packets and applies some action to them. When multiple rules are specified, they are executed in order. | Field | Description | Accepted Values | Schema | Default | | ----------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------ | ----------------------------- | ------- | | metadata | Per-rule metadata. | | [RuleMetadata](#rulemetadata) | | | action | Action to perform when matching this rule. | `Allow`, `Deny`, `Log`, `Pass` | string | | | protocol | Positive protocol match. | `TCP`, `UDP`, `ICMP`, `ICMPv6`, `SCTP`, `UDPLite`, `1`-`255` | string \| integer | | | notProtocol | Negative protocol match. | `TCP`, `UDP`, `ICMP`, `ICMPv6`, `SCTP`, `UDPLite`, `1`-`255` | string \| integer | | | icmp | ICMP match criteria. | | [ICMP](#icmp) | | | notICMP | Negative match on ICMP. | | [ICMP](#icmp) | | | ipVersion | Positive IP version match. | `4`, `6` | integer | | | source | Source match parameters. | | [EntityRule](#entityrule) | | | destination | Destination match parameters. | | [EntityRule](#entityrule) | | | http | Match HTTP request parameters. Application layer policy must be enabled to use this field. | | [HTTPMatch](#httpmatch) | | After a `Log` action, processing continues with the next rule; `Allow` and `Deny` are immediate and final and no further rules are processed. An `action` of `Pass` in a `NetworkPolicy` or `GlobalNetworkPolicy` will skip over the remaining policies and jump to the first profile assigned to the endpoint, applying the policy configured in the profile; if there are no Profiles configured for the endpoint the default applied action is `Deny`. ### RuleMetadata[​](#rulemetadata) Metadata associated with a specific rule (rather than the policy as a whole). The contents of the metadata does not affect how a rule is interpreted or enforced; it is simply a way to store additional information for use by operators or applications that interact with Calico Cloud. | Field | Description | Schema | Default | | ----------- | ----------------------------------- | ----------------------- | ------- | | annotations | Arbitrary non-identifying metadata. | map of string to string | | Example: ```yaml metadata: annotations: app: database owner: devops ``` Annotations follow the [same rules as Kubernetes for valid syntax and character set](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set). On Linux with the iptables data plane, rule annotations are rendered as comments in the form `-m comment --comment "="` on the iptables rule(s) that correspond to the Calico Cloud rule. ### ICMP[​](#icmp) | Field | Description | Accepted Values | Schema | Default | | ----- | ------------------- | -------------------- | ------- | ------- | | type | Match on ICMP type. | Can be integer 0-254 | integer | | | code | Match on ICMP code. | Can be integer 0-255 | integer | | ### EntityRule[​](#entityrule) Entity rules specify the attributes of the source or destination of a packet that must match for the rule as a whole to match. Packets can be matched on combinations of: - Identity of the source/destination, by using [Selectors](#selectors) or by specifying a particular Kubernetes `Service`. Selectors can match [workload endpoints](https://docs.tigera.io/calico-cloud/reference/resources/workloadendpoint), [host endpoints](https://docs.tigera.io/calico-cloud/reference/resources/hostendpoint) and ([namespaced](https://docs.tigera.io/calico-cloud/reference/resources/networkset) or [global](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkset)) network sets. - Source/destination IP address, protocol and port. If the rule contains multiple match criteria (for example, an IP and a port) then all match criteria must match for the rule as a whole to match a packet. | Field | Description | Accepted Values | Schema | Default | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------------------------- | ------- | | nets | Match packets with IP in any of the listed CIDRs. | List of valid IPv4 CIDRs or list of valid IPv6 CIDRs (IPv4 and IPv6 CIDRs shouldn't be mixed in one rule) | list of cidrs | | | notNets | Negative match on CIDRs. Match packets with IP not in any of the listed CIDRs. | List of valid IPv4 CIDRs or list of valid IPv6 CIDRs (IPv4 and IPv6 CIDRs shouldn't be mixed in one rule) | list of cidrs | | | selector | Positive match on selected endpoints. If a `namespaceSelector` is also defined, the set of endpoints this applies to is limited to the endpoints in the selected namespaces. | Valid selector | [selector](#selector) | | | notSelector | Negative match on selected endpoints. If a `namespaceSelector` is also defined, the set of endpoints this applies to is limited to the endpoints in the selected namespaces. | Valid selector | [selector](#selector) | | | namespaceSelector | Positive match on selected namespaces. If specified, only workload endpoints in the selected Kubernetes namespaces are matched. Matches namespaces based on the labels that have been applied to the namespaces. Defines the scope that selectors will apply to, if not defined then selectors apply to the NetworkPolicy's namespace. Match a specific namespace by name using the `projectcalico.org/name` label. Select the non-namespaced resources like GlobalNetworkSet(s), host endpoints to which this policy applies by using `global()` selector. | Valid selector | [selector](#selector) | | | ports | Positive match on the specified ports | | list of [ports](#ports) | | | domains | Positive match on [domain names](#exact-and-wildcard-domain-names). | List of [exact or wildcard domain names](#exact-and-wildcard-domain-names) | list of strings | | | notPorts | Negative match on the specified ports | | list of [ports](#ports) | | | serviceAccounts | Match endpoints running under service accounts. If a `namespaceSelector` is also defined, the set of service accounts this applies to is limited to the service accounts in the selected namespaces. | | [ServiceAccountMatch](#serviceaccountmatch) | | | services | Match the specified service(s). If specified on egress rule destinations, no other selection criteria can be set. If specified on ingress rule sources, only positive or negative matches on ports can be specified. | | [ServiceMatch](#servicematch) | | > **SECONDARY:** You cannot mix IPv4 and IPv6 CIDRs in a single rule using `nets` or `notNets`. If you need to match both, create 2 rules. #### Selector performance in EntityRules[​](#selector-performance-in-entityrules) When rendering policy into the data plane, Calico Cloud must identify the endpoints that match the selectors in all active rules. This calculation is optimized for certain common selector types. Using the optimized selector types reduces CPU usage (and policy rendering time) by orders of magnitude. This becomes important at high scale (hundreds of active rules, hundreds of thousands of endpoints). The optimized operators are as follows: - `label == "value"` - `label in { 'v1', 'v2' }` - `has(label)` - ` && ` is optimized if **either** `` or `` is optimized. The following perform like `has(label)`. All endpoints with the label will be scanned to find matches: - `label contains 's'` - `label starts with 's'` - `label ends with 's'` The other operators, and in particular, `all()`, `!`, `||` and `!=` are not optimized. Examples: - `a == 'b'` - optimized - `a == 'b' && has(c)` - optimized - `a == 'b' || has(c)` - **not** optimized due to use of `||` - `c != 'd'` - **not** optimized due to use of `!=` - `!has(a)` - **not** optimized due to use of `!` - `a == 'b' && c != 'd'` - optimized, `a =='b'` is optimized so `a == 'b' && ` is optimized. - `c != 'd' && a == 'b'` - optimized, `a =='b'` is optimized so ` && a == 'b'` is optimized. ### Exact and wildcard domain names[​](#exact-and-wildcard-domain-names) The `domains` field is only valid for egress Allow rules. It restricts the rule to apply only to traffic to one of the specified domains. If this field is specified, the parent [Rule](#rule)'s `action` must be `Allow`, and `nets` and `selector` must both be left empty. When a configured domain name has no wildcard (`*`), it matches exactly that domain name. For example: - `microsoft.com` - `tigera.io` With a single asterisk in any part of the domain name, it matches 1 or more path components at that position. For example: - `*.google.com` matches `www.google.com` and `www.ipv6.google.com`, but not `google.com` - `www.*.com` matches `www.sun.com` and `www.apple.com`, but not `www.com` - `update.*.mycompany.com` matches `update.tools.mycompany.com`, `update.secure.suite.mycompany.com`, and so on **Not** supported are: - Multiple wildcards in the same domain, for example: `*.*.mycompany.com` - Asterisks that are not the entire component, for example: `www.g*.com` - A wildcard as the last component, for example: `www.mycompany.*` - More general wildcards, such as regular expressions > **SECONDARY:** Calico Cloud implements policy for domain names by learning the corresponding IPs from DNS, then programming rules to allow those IPs. This means that if multiple domain names A, B and C all map to the same IP, and there is domain-based policy to allow A, traffic to B and C will be allowed as well. ### Selector[​](#selector) A label selector is an expression which either matches or does not match a resource based on its labels. Calico Cloud label selectors support a number of operators, which can be combined into larger expressions using the boolean operators and parentheses. | Expression | Meaning | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Logical operators** | | | `( )` | Matches if and only if `` matches. (Parentheses are used for grouping expressions.) | | `! ` | Matches if and only if `` does not match. **Tip:** `!` is a special character at the start of a YAML string, if you need to use `!` at the start of a YAML string, enclose the string in quotes. | | ` && ` | "And": matches if and only if both ``, and, `` matches | | ` \|\| ` | "Or": matches if and only if either ``, or, `` matches. | | **Match operators** | | | `all()` | Match all in-scope resources. To match *no* resources, combine this operator with `!` to form `!all()`. | | `global()` | Match all non-namespaced resources. Useful in a `namespaceSelector` to select global resources such as global network sets. | | `k == 'v'` | Matches resources with the label 'k' and value 'v'. | | `k != 'v'` | Matches resources without label 'k' or with label 'k' and value *not* equal to `v` | | `has(k)` | Matches resources with label 'k', independent of value. To match pods that do not have label `k`, combine this operator with `!` to form `!has(k)` | | `k in { 'v1', 'v2' }` | Matches resources with label 'k' and value in the given set | | `k not in { 'v1', 'v2' }` | Matches resources without label 'k' or with label 'k' and value *not* in the given set | | `k contains 's'` | Matches resources with label 'k' and value containing the substring 's' | | `k starts with 's'` | Matches resources with label 'k' and value starting with the substring 's' | | `k ends with 's'` | Matches resources with label 'k' and value ending with the substring 's' | Operators have the following precedence: - **Highest**: all the match operators - Parentheses `( ... )` - Negation with `!` - Conjunction with `&&` - **Lowest**: Disjunction with `||` For example, the expression ```text ! has(my-label) || my-label starts with 'prod' && role in {'frontend','business'} ``` Would be "bracketed" like this: ```text ((!(has(my-label)) || ((my-label starts with 'prod') && (role in {'frontend','business'})) ``` It would match: - Any resource that did not have label "my-label". - Any resource that both: - Has a value for `my-label` that starts with "prod", and, - Has a role label with value either "frontend", or "business". Understanding scopes and the `all()` and `global()` operators: selectors have a scope of resources that they are matched against, which depends on the context in which they are used. For example: - The `nodeSelector` in an `IPPool` selects over `Node` resources. - The top-level selector in a `NetworkPolicy` selects over the workloads *in the same namespace* as the `NetworkPolicy`. - The top-level selector in a `GlobalNetworkPolicy` doesn't have the same restriction, it selects over all endpoints including namespaced `WorkloadEndpoint`s and non-namespaced `HostEndpoint`s. - The `namespaceSelector` in a `NetworkPolicy` (or `GlobalNetworkPolicy`) *rule* selects over the labels on namespaces rather than workloads. - The `namespaceSelector` determines the scope of the accompanying `selector` in the entity rule. If no `namespaceSelector` is present then the rule's `selector` matches the default scope for that type of policy. (This is the same namespace for `NetworkPolicy` and all endpoints/network sets for `GlobalNetworkPolicy`) - The `global()` operator can be used (only) in a `namespaceSelector` to change the scope of the main `selector` to include non-namespaced resources such as [GlobalNetworkSet](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkset). This allows namespaced `NetworkPolicy` resources to refer to global non-namespaced resources, which would otherwise be impossible. ### Ports[​](#ports) Calico Cloud supports the following syntaxes for expressing ports. | Syntax | Example | Description | | --------- | ---------- | ------------------------------------------------------------------- | | int | 80 | The exact (numeric) port specified | | start:end | 6040:6050 | All (numeric) ports within the range start ≤ x ≤ end | | string | named-port | A named port, as defined in the ports list of one or more endpoints | An individual numeric port may be specified as a YAML/JSON integer. A port range or named port must be represented as a string. For example, this would be a valid list of ports: ```yaml ports: [8080, '1234:5678', 'named-port'] ``` #### Named ports[​](#named-ports) Using a named port in an `EntityRule`, instead of a numeric port, gives a layer of indirection, allowing for the named port to map to different numeric values for each endpoint. For example, suppose you have multiple HTTP servers running as workloads; some exposing their HTTP port on port 80 and others on port 8080. In each workload, you could create a named port called `http-port` that maps to the correct local port. Then, in a rule, you could refer to the name `http-port` instead of writing a different rule for each type of server. > **SECONDARY:** Since each named port may refer to many endpoints (and Calico Cloud has to expand a named port into a set of endpoint/port combinations), using a named port is considerably more expensive in terms of CPU than using a simple numeric port. We recommend that they are used sparingly, only where the extra indirection is required. ### ServiceAccountMatch[​](#serviceaccountmatch) A ServiceAccountMatch matches service accounts in an EntityRule. | Field | Description | Schema | | -------- | ------------------------------- | --------------------- | | names | Match service accounts by name | list of strings | | selector | Match service accounts by label | [selector](#selector) | ### ServiceMatch[​](#servicematch) A ServiceMatch matches a service in an EntityRule. | Field | Description | Schema | | --------- | ------------------------ | ------ | | name | The service's name. | string | | namespace | The service's namespace. | string | ### Performance Hints[​](#performance-hints) Performance hints provide a way to tell Calico Cloud about the intended use of the policy so that it may process it more efficiently. Currently only one hint is defined: - `AssumeNeededOnEveryNode`: normally, Calico Cloud only calculates a policy's rules and selectors on nodes where the policy is actually in use (i.e. its selector matches a local endpoint). This saves work in most cases. The `AssumeNeededOnEveryNode` hint tells Calico Cloud to treat the policy as "in use" on *every* node. This is useful for large policy sets that are known to apply to all (or nearly all) endpoints. It effectively "preloads" the policy on every node so that there is less work to do when the first endpoint matching the policy shows up. It also prevents work from being done to tear down the policy when the last endpoint is drained. ## Application layer policy[​](#application-layer-policy) Application layer policy is an optional feature of Calico Cloud and [must be enabled](https://docs.tigera.io/calico-cloud/network-policy/application-layer-policies/alp) to use the following match criteria. > **SECONDARY:** Application layer policy match criteria are supported with the following restrictions. > > - Only ingress policy is supported. Egress policy must not contain any application layer policy match clauses. > - Rules must have the action `Allow` if they contain application layer policy match clauses. ### HTTPMatch[​](#httpmatch) An HTTPMatch matches attributes of an HTTP request. The presence of an HTTPMatch clause on a Rule will cause that rule to only match HTTP traffic. Other application layer protocols will not match the rule. Example: ```yaml http: methods: ['GET', 'PUT'] paths: - exact: '/projects/calico' - prefix: '/users' headers: - header: 'x-forwarded-for' operator: 'HasPrefix' values: ['192.168.0.1', '192.168.0.254'] ``` | Field | Description | Schema | | ------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- | | methods | Match HTTP methods. Case sensitive. [Standard HTTP method descriptions.](https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html) | list of strings | | paths | Match HTTP paths. Case sensitive. | list of [HTTPPathMatch](#httppathmatch) | | headers | Match HTTP headers. | list of [HTTPHeaderMatch](#httpheadermatch) | ### HTTPPathMatch[​](#httppathmatch) | Syntax | Example | Description | | ------ | ------------------- | ------------------------------------------------------------------------------- | | exact | `exact: "/foo/bar"` | Matches the exact path as written, not including the query string or fragments. | | prefix | `prefix: "/keys"` | Matches any path that begins with the given prefix. | ### HTTPHeaderMatch[​](#httpheadermatch) | Syntax | Example | Description | | -------- | ---------------------------------- | ------------------------------------------------------------------------------------------ | | header | `x-forwarded-for` | Name of a HTTP header. Header names are case insensitive, please use lowercase characters. | | operator | `In` | Operator name to apply to the HTTP header value. Case sensitive. | | values | `['192.168.0.1', '192.168.0.254']` | Values that the operator will test the HTTP header value against. Case sensitive. | The following operators are allowed: - `Exists`: matches the HTTP request header if the specified header exists. `values` are ignored. - `DoesNotExist`: matches the HTTP request header if the specified header does not exist. `values` are ignored. - `HasPrefix`: matches the HTTP request header if the specified header has a prefix from any the values provided. - `HasSuffix`: matches the HTTP request header if the specified header has a suffix from any the values provided. - `In`: matches the HTTP request header if its value is in the set of values provided. - `NotIn`: matches the HTTP request header if its value is not in the set of values provided. - `MatchesRegex`: matches the HTTP request header if its value matches any of the regular expressions in values field. ## Supported operations[​](#supported-operations) | Datastore type | Create/Delete | Update | Get/List | Notes | | ------------------------ | ------------- | ------ | -------- | ----- | | Kubernetes API datastore | Yes | Yes | Yes | | #### List filtering on tiers[​](#list-filtering-on-tiers) List and watch operations may specify label selectors or field selectors to filter `GlobalNetworkPolicy` resources on tiers returned by the API server. When no selector is specified, the API server returns all `GlobalNetworkPolicy` resources from all tiers that the user has access to. ##### Field selector[​](#field-selector) When using the field selector, supported operators are `=` and `==` The following example shows how to retrieve all `GlobalNetworkPolicy` resources in the default tier: ```bash kubectl get globalnetworkpolicy --field-selector spec.tier=default ``` ##### Label selector[​](#label-selector) When using the label selector, supported operators are `=`, `==` and `IN`. The following example shows how to retrieve all `GlobalNetworkPolicy` resources in the `default` and `net-sec` tiers: ```bash kubectl get globalnetworkpolicy -l 'projectcalico.org/tier in (default, net-sec)' ``` ### Global network set A global network set resource (GlobalNetworkSet) represents an arbitrary set of IP subnetworks/CIDRs, allowing it to be matched by Calico Cloud policy. Network sets are useful for applying policy to traffic coming from (or going to) external, non-Calico Cloud, networks. GlobalNetworkSets can also include domain names, whose effect is to allow egress traffic to those domain names, when the GlobalNetworkSet is matched by the destination selector of an egress rule with action Allow. Domain names have no effect in ingress rules, or in a rule whose action is not Allow. > **SECONDARY:** Calico Cloud implements policy for domain names by learning the corresponding IPs from DNS, then programming rules to allow those IPs. This means that if multiple domain names A, B and C all map to the same IP, and there is domain-based policy to allow A, traffic to B and C will be allowed as well. The metadata for each network set includes a set of labels. When Calico Cloud is calculating the set of IPs that should match a source/destination selector within a [global network policy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) rule, or within a [network policy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) rule whose `namespaceSelector` includes `global()`, it includes the CIDRs from any network sets that match the selector. > **SECONDARY:** Since Calico Cloud matches packets based on their source/destination IP addresses, Calico Cloud rules may not behave as expected if there is NAT between the Calico Cloud-enabled node and the networks listed in a network set. For example, in Kubernetes, incoming traffic via a service IP is typically SNATed by the kube-proxy before reaching the destination host so Calico Cloud's workload policy will see the kube-proxy's host's IP as the source instead of the real source. For `kubectl` [commands](https://kubernetes.io/docs/reference/kubectl/overview/), the following case-insensitive aliases may be used to specify the resource type on the CLI: `globalnetworkset.projectcalico.org`, `globalnetworksets.projectcalico.org` and abbreviations such as `globalnetworkset.p` and `globalnetworksets.p`. ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: GlobalNetworkSet metadata: name: a-name-for-the-set labels: role: external-database spec: nets: - 198.51.100.0/28 - 203.0.113.0/24 allowedEgressDomains: - db.com - '*.db.com' ``` ## Global network set definition[​](#global-network-set-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | | ------ | ------------------------------------------ | ------------------------------------------------- | ------ | | name | The name of this network set. | Lower-case alphanumeric with optional `-` or `-`. | string | | labels | A set of labels to apply to this endpoint. | | map | ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------- | ------ | ------- | | nets | The IP networks/CIDRs to include in the set. | Valid IPv4 or IPv6 CIDRs, for example "192.0.2.128/25" | list | | | allowedEgressDomains | The list of domain names that belong to this set and are honored in egress allow rules only. Domain names specified here only work to allow egress traffic from the cluster to external destinations. They don't work to *deny* traffic to destinations specified by domain name, or to allow ingress traffic from *sources* specified by domain name. | List of [exact or wildcard domain names](#exact-and-wildcard-domain-names) | list | | ### Exact and wildcard domain names[​](#exact-and-wildcard-domain-names) When a configured domain name has no wildcard (`*`), it matches exactly that domain name. For example: - `microsoft.com` - `tigera.io` With a single asterisk in any part of the domain name, it matches 1 or more path components at that position. For example: - `*.google.com` matches `www.google.com` and `www.ipv6.google.com`, but not `google.com` - `www.*.com` matches `www.sun.com` and `www.apple.com`, but not `www.com` - `update.*.mycompany.com` matches `update.tools.mycompany.com`, `update.secure.suite.mycompany.com`, and so on **Not** supported are: - Multiple wildcards in the same domain, for example: `*.*.mycompany.com` - Asterisks that are not the entire component, for example: `www.g*.com` - A wildcard as the last component, for example: `www.mycompany.*` - More general wildcards, such as regular expressions ### Global report A global report resource is a configuration for generating compliance reports. A global report configuration in Calico Cloud lets you: - Specify report contents, frequency, and data filtering - Specify the node(s) on which to run the report generation jobs - Enable/disable creation of new jobs for generating the report For `kubectl` [commands](https://kubernetes.io/docs/reference/kubectl/overview/), the following case-insensitive aliases may be used to specify the resource type on the CLI: `globalreport.projectcalico.org`, `globalreports.projectcalico.org` and abbreviations such as `globalreport.p` and `globalreports.p`. ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: GlobalReport metadata: name: weekly-full-inventory spec: reportType: inventory schedule: 0 0 * * 0 jobNodeSelector: nodetype: infrastructure --- apiVersion: projectcalico.org/v3 kind: GlobalReport metadata: name: hourly-accounts-networkaccess spec: reportType: network-access endpoints: namespaces: names: ['payable', 'collections', 'payroll'] schedule: 0 * * * * --- apiVersion: projectcalico.org/v3 kind: GlobalReport metadata: name: monthly-widgets-controller-tigera-policy-audit spec: reportType: policy-audit schedule: 0 0 1 * * endpoints: serviceAccounts: names: ['controller'] namespaces: names: ['widgets'] --- apiVersion: projectcalico.org/v3 kind: GlobalReport metadata: name: daily-cis-benchmark spec: reportType: cis-benchmark schedule: 0 0 * * * cis: resultsFilters: - benchmarkSelection: { kubernetesVersion: '1.13' } exclude: ['1.1.4', '1.2.5'] ``` ## GlobalReport Definition[​](#globalreport-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | | ------ | ---------------------------------------- | ------------------------------------------------ | ------ | | name | The name of this report. | Lower-case alphanumeric with optional `-` or `.` | string | | labels | A set of labels to apply to this report. | | map | ### Spec[​](#spec) | Field | Description | Required | Accepted Values | Schema | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- | | reportType | The type of report to produce. This field controls the content of the report - see the links for each type for more details. | Yes | [cis‑benchmark](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/cis-benchmark), [inventory](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/inventory), [network‑access](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/network-access), [policy‑audit](https://docs.tigera.io/calico-cloud/reference/resources/compliance-reports/policy-audit) | string | | endpoints | Specify which endpoints are in scope. If omitted, selects everything. | | | [EndpointsSelection](#endpointsselection) | | schedule | Configure report frequency by specifying start and end time in [cron-format](https://en.wikipedia.org/wiki/Cron). Reports are started 30 minutes (configurable) after the scheduled value to allow enough time for data archival. A maximum limit of 12 schedules per hour is enforced (an average of one report every 5 minutes). | Yes | | string | | jobNodeSelector | Specify the node(s) for scheduling the report jobs using selectors. | | | map | | suspend | Disable future scheduled report jobs. In-flight reports are not affected. | | | bool | | cis | Parameters related to generating a CIS benchmark report. | | | [CISBenchmarkParams](#cisbenchmarkparams) | ### EndpointsSelection[​](#endpointsselection) | Field | Description | Schema | | --------------- | ------------------------------------------------------------------------------------------- | ------------------------------------------- | | selector | Endpoint label selector to restrict endpoint selection. | string | | namespaces | Namespace name and label selector to restrict endpoints by selected namespaces. | [NamesAndLabelsMatch](#namesandlabelsmatch) | | serviceAccounts | Service account name and label selector to restrict endpoints by selected service accounts. | [NamesAndLabelsMatch](#namesandlabelsmatch) | ### CISBenchmarkParams[​](#cisbenchmarkparams) | Fields | Description | Required | Schema | | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ----------------------------------------- | | highThreshold | Integer percentage value that determines the lower limit of passing tests to consider a node as healthy. Default: 100 | No | int | | medThreshold | Integer percentage value that determines the lower limit of passing tests to consider a node as unhealthy. Default: 50 | No | int | | includeUnscoredTests | Boolean value that when false, applies a filter to exclude tests that are marked as “Unscored” by the CIS benchmark standard. If true, the tests will be included in the report. Default: false | No | bool | | numFailedTests | Integer value that sets the number of tests to display in the Top-failed Tests section of the CIS benchmark report. Default: 5 | No | int | | resultsFilters | Specifies an include or exclude filter to apply on the test results that will appear on the report. | No | [CISBenchmarkFilter](#cisbenchmarkfilter) | ### CISBenchmarkFilter[​](#cisbenchmarkfilter) | Fields | Description | Required | Schema | | ------------------ | ---------------------------------------------------------------------------------------------- | -------- | ----------------------------------------------- | | benchmarkSelection | Specify which set of benchmarks that this filter should apply to. Selects all benchmark types. | No | [CISBenchmarkSelection](#cisbenchmarkselection) | | exclude | Specify which benchmark tests to exclude | No | array of strings | | include | Specify which benchmark tests to include only (higher precedence than exclude) | No | array of strings | ### CISBenchmarkSelection[​](#cisbenchmarkselection) | Fields | Description | Required | Schema | | ----------------- | -------------------------------------- | -------- | ------ | | kubernetesVersion | Specifies a version of the benchmarks. | Yes | string | ### NamesAndLabelsMatch[​](#namesandlabelsmatch) | Field | Description | Schema | | -------- | ------------------------------------ | ------ | | names | Set of resource names. | list | | selector | Selects a set of resources by label. | string | Use the `NamesAndLabelsMatch`to limit the scope of endpoints. If both `names` and `selector` are specified, the resource is identified using label *AND* name match. > **SECONDARY:** To use the Calico Cloud compliance reporting feature, you must ensure all required resource types are being audited and the logs archived in Elasticsearch. You must explicitly configure the [Kubernetes API Server](https://docs.tigera.io/calico-cloud/observability/kube-audit) to send audit logs for Kubernetes-owned resources to Elasticsearch. ## Supported operations[​](#supported-operations) | Datastore type | Create/Delete | Update | Get/List | Notes | | --------------------- | ------------- | ------ | -------- | ----- | | Kubernetes API server | Yes | Yes | Yes | | ### Global threat feed A global threat feed resource (GlobalThreatFeed) represents a feed of threat intelligence used for security purposes. Calico Cloud supports threat feeds that give either - a set of IP addresses or IP prefixes, with content type IPSet, or - a set of domain names, with content type DomainNameSet For each IPSet threat feed, Calico Cloud automatically monitors flow logs for members of the set. IPSet threat feeds can also be configured to be synchronized to a [global network set](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkset), allowing you to use them as a dynamically-updating deny-list by incorporating the global network set into network policy. For each DomainNameSet threat feed, Calico Cloud automatically monitors DNS logs for queries (QNAME) or answers (RR NAME or RDATA) that contain members of the set. For `kubectl` [commands](https://kubernetes.io/docs/reference/kubectl/overview/), the following case-insensitive aliases may be used to specify the resource type on the CLI: `globalthreatfeed.projectcalico.org`, `globalthreatfeeds.projectcalico.org` and abbreviations such as `globalthreatfeed.p` and `globalthreatfeeds.p`. ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: GlobalThreatFeed metadata: name: sample-global-threat-feed spec: content: IPSet mode: Enabled description: "This is the sample global threat feed" feedType: Custom globalNetworkSet: # labels to set on the GNS labels: level: high pull: # accepts time in golang duration format period: 24h http: format: newlineDelimited: {} url: https://an.example.threat.feed/deny-list headers: - name: "Accept" value: "text/plain" - name: "APIKey" valueFrom: # secrets selected must be in the "tigera-intrusion-detection" namespace to be used secretKeyRef: name: "globalthreatfeed-sample-global-threat-feed-example" key: "apikey" ``` ## Push or Pull[​](#push-or-pull) You can configure Calico Cloud to pull updates from your threat feed using a [`pull`](#pull) stanza in the global threat feed spec. Alternately, you can have your threat feed push updates directly. Leave out the `pull` stanza, and configure your threat feed to create or update the Elasticsearch document that corresponds to the global threat feed object. For IPSet threat feeds, this Elasticsearch document will be in the index `.tigera.ipset.` and must have the ID set to the name of the global threat feed object. The doc should have a single field called `ips`, containing a list of IP prefixes. For example: ```text PUT .tigera.ipset.cluster01/_doc/sample-global-threat-feed { "ips" : ["99.99.99.99/32", "100.100.100.0/24"] } ``` For DomainNameSet threat feeds, this Elasticsearch document will be in the index `.tigera.domainnameset.` and must have the ID set to the name of the global threat feed object. The doc should have a single field called `domains`, containing a list of domain names. For example: ```text PUT .tigera.domainnameset.cluster01/_doc/example-global-threat-feed { "domains" : ["malware.badstuff", "hackers.r.us"] } ``` Refer to the [Elasticsearch document APIs](https://www.elastic.co/guide/en/elasticsearch/reference/6.4/docs-update.html) for more information on how to create and update documents in Elasticsearch. ## GlobalThreatFeed Definition[​](#globalthreatfeed-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | | ------ | --------------------------------------------- | ----------------------------------------- | ------ | | name | The name of this threat feed. | Lower-case alphanumeric with optional `-` | string | | labels | A set of labels to apply to this threat feed. | | map | ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | ---------------- | ---------------------------------------------------- | ---------------------- | --------------------------------------------- | ------- | | content | What kind of threat intelligence is provided | IPSet, DomainNameSet | string | IPSet | | mode | Determines if the threat feed is Enabled or Disabled | Enabled, Disabled | string | Enabled | | description | Human-readable description of the template | Maximum 256 characters | string | | | feedType | Distinguishes Builtin threat feeds from Custom feeds | Builtin, Custom | string | Custom | | globalNetworkSet | Include to sync with a global network set | | [GlobalNetworkSetSync](#globalnetworksetsync) | | | pull | Configure periodic pull of threat feed updates | | [Pull](#pull) | | ### Status[​](#status) The `status` is read-only for users and updated by the `intrusion-detection-controller` component as it processes global threat feeds. | Field | Description | | -------------------- | -------------------------------------------------------------------------------- | | lastSuccessfulSync | Timestamp of the last successful update to the threat intelligence from the feed | | lastSuccessfulSearch | Timestamp of the last successful search of logs for threats | | errorConditions | List of errors preventing operation of the updates or search | ### GlobalNetworkSetSync[​](#globalnetworksetsync) When you include a `globalNetworkSet` stanza in a global threat feed, it triggers synchronization with a [global network set](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkset). This global network set will have the name `threatfeed.` where `` is the name of the global threat feed it is synced with. This is only supported for threat feeds of type IPSet. > **SECONDARY:** A `globalNetworkSet` stanza only works for `IPSet` threat feeds, and you must also include a `pull` stanza. | Field | Description | Accepted Values | Schema | | ------ | --------------------------------------------------------- | --------------- | ------ | | labels | A set of labels to apply to the synced global network set | | map | ### Pull[​](#pull) When you include a `pull` stanza in a global threat feed, it triggers a periodic pull of new data. On successful pull and update to the data store, we update the `status.lastSuccessfulSync` timestamp. If you do not include a `pull` stanza, you must configure your system to [push](#push-or-pull) updates. | Field | Description | Accepted Values | Schema | Default | | ------ | ------------------------------------- | --------------- | ------------------------------------------------------------- | ------- | | period | How often to pull an update | ≥ 5m | [Duration string](https://golang.org/pkg/time/#ParseDuration) | 24h | | http | Pull the update from an HTTP endpoint | | [HTTPPull](#httppull) | | ### HTTPPull[​](#httppull) Pull updates from the threat feed by doing an HTTP GET against the given URL. | Field | Description | Accepted Values | Schema | | ------- | --------------------------------------------------------- | --------------- | ------------------------- | | format | Format of the data the threat feed returns | | [Format](#format) | | url | The URL to query | | string | | headers | List of additional HTTP Headers to include on the request | | [HTTPHeader](#httpheader) | IPSet threat feeds must contain IP addresses or IP prefixes. For example: ```text This is an IP Prefix 100.100.100.0/24 This is an address 99.99.99.99 ``` DomainNameSet threat feeds must contain domain names. For example: ```text Suspicious domains malware.badstuff hackers.r.us ``` Internationalized domain names (IDNA) may be encoded either as Unicode in UTF-8 format, or as ASCII-Compatible Encoding (ACE) according to [RFC 5890](https://tools.ietf.org/html/rfc5890). ### Format[​](#format) Several different feed formats are supported. The default, `newlineDelimited`, expects a text file containing entries separated by newline characters. It may also include comments prefixed by `#`. `json` uses a [jsonpath](https://goessner.net/articles/JsonPath/) to extract the desired information from a JSON document. `csv` extracts one column from CSV-formatted data. | Field | Description | Schema | | ---------------- | --------------------------- | ------------- | | newlineDelimited | Newline-delimited text file | Empty object | | json | JSON object | [JSON](#json) | | csv | CSV file | [CSV](#csv) | #### JSON[​](#json) | Field | Description | Schema | | ----- | ---------------------------------------------------------------------- | ------ | | path | [jsonpath](https://goessner.net/articles/JsonPath/) to extract values. | string | Values can be extracted from the document using any [jsonpath](https://goessner.net/articles/JsonPath/) expression, subject to the limitations mentioned below, that evaluates to a list of strings. For example: `$.` is valid for `["a", "b", "c"]`, and `$.a` is valid for `{"a": ["b", "c"]}`. > **WARNING:** No support for subexpressions and filters. Strings in brackets must use double quotes. It cannot operate on JSON decoded struct fields. #### CSV[​](#csv) \| Field | Description | Schema | | --------------------------- | ------------------------------------------------------------------------- | ------ | ------ | | fieldNum | Number of column containing values. Mutually exclusive with `fieldName`. | int | | fieldName | Name of column containing values, requires `header: true`. | string | | header | Whether or not the document contains a header row. | bool | | columnDelimiter | An alternative delimiter character, such as `|`. | string | | commentDelimiter | Lines beginning with this character are skipped. `#` is common. | string | | recordSize | The number of columns expected in the document. Auto detected if omitted. | int | | disableRecordSizeValidation | Disable row size checking. Mutually exclusive with `recordSize`. | bool | ### HTTPHeader[​](#httpheader) | Field | Description | Schema | | --------- | --------------------------------------------------------- | ------------------------------------- | | name | Header name | string | | value | Literal value | string | | valueFrom | Include to retrieve the value from a config map or secret | [HTTPHeaderSource](#httpheadersource) | > **SECONDARY:** You must include either `value` or `valueFrom`, but not both. ### HTTPHeaderSource[​](#httpheadersource) | Field | Description | Schema | | --------------- | ------------------------------- | ----------------- | | configMapKeyRef | Get the value from a config map | [KeyRef](#keyref) | | secretKeyRef | Get the value from a secret | [KeyRef](#keyref) | ### KeyRef[​](#keyref) KeyRef tells Calico Cloud where to get the value for a header. The referenced Kubernetes object (either a config map or a secret) must be in the `tigera-intrusion-detection` namespace. The referenced Kubernetes object should have a name with following prefix format: `globalthreatfeed--`. | Field | Description | Accepted Values | Schema | Default | | -------- | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | ------ | ------- | | name | The name of the config map or secret | | string | | | key | The key within the config map or secret | | string | | | optional | Whether the pull can proceed without the referenced value | If the referenced value does not exist, `true` means omit the header. `false` means abort the entire pull until it exists | bool | `false` | ### Host endpoint A host endpoint resource (`HostEndpoint`) represents one or more real or virtual interfaces attached to a host that is running Calico Cloud. It enforces Calico Cloud policy on the traffic that is entering or leaving the host's default network namespace through those interfaces. - A host endpoint with `interfaceName: *` represents *all* of a host's real or virtual interfaces. - A host endpoint for one specific real interface is configured by `interfaceName: `, for example `interfaceName: eth0`, or by leaving `interfaceName` empty and including one of the interface's IPs in `expectedIPs`. Each host endpoint may include a set of labels and list of profiles that Calico Cloud will use to apply [policy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) to the interface. If no profiles or labels are applied, Calico Cloud will not apply any policy. For `kubectl` [commands](https://kubernetes.io/docs/reference/kubectl/overview/), the following case-insensitive aliases may be used to specify the resource type on the CLI: `hostendpoint.projectcalico.org`, `hostendpoints.projectcalico.org` and abbreviations such as `hostendpoint.p` and `hostendpoints.p`. **Default behavior of external traffic to/from host** If a host endpoint is created and network policy is not in place, the Calico Cloud default is to deny traffic to/from that endpoint (except for traffic allowed by failsafe rules). For a named host endpoint (i.e. a host endpoint representing a specific interface), Calico Cloud blocks traffic only to/from the interface specified in the host endpoint. Traffic to/from other interfaces is ignored. > **SECONDARY:** Host endpoints with `interfaceName: *` do not support [untracked policy](https://docs.tigera.io/calico-cloud/network-policy/extreme-traffic/high-connection-workloads). For a wildcard host endpoint (i.e. a host endpoint representing all of a host's interfaces), Calico Cloud blocks traffic to/from *all* interfaces on the host (except for traffic allowed by failsafe rules). However, profiles can be used in conjunction with host endpoints to modify default behavior of external traffic to/from the host in the absence of network policy. Calico Cloud provides a default profile resource named `projectcalico-default-allow` that consists of allow-all ingress and egress rules. Host endpoints with the `projectcalico-default-allow` profile attached will have "allow-all" semantics instead of "deny-all" in the absence of policy. Note: If you have custom iptables rules, using host endpoints with allow-all rules (with no policies) will accept all traffic and therefore bypass those custom rules. > **SECONDARY:** Auto host endpoints specify the `projectcalico-default-allow` profile so they behave similarly to pod workload endpoints. > **SECONDARY:** When rendering security rules on other hosts, Calico Cloud uses the `expectedIPs` field to resolve label selectors to IP addresses. If the `expectedIPs` field is omitted then security rules that use labels will fail to match this endpoint. **Host to local workload traffic**: Traffic from a host to its workload endpoints (e.g. Kubernetes pods) is always allowed, despite any policy in place. This ensures that `kubelet` liveness and readiness probes always work. ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: HostEndpoint metadata: name: some.name labels: type: production spec: interfaceName: eth0 node: myhost expectedIPs: - 192.168.0.1 - 192.168.0.2 profiles: - profile1 - profile2 ports: - name: some-port port: 1234 protocol: TCP - name: another-port port: 5432 protocol: UDP ``` ## Host endpoint definition[​](#host-endpoint-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | | ------ | ------------------------------------------ | --------------------------------------------------- | ------ | | name | The name of this hostEndpoint. Required. | Alphanumeric string with optional `.`, `_`, or `-`. | string | | labels | A set of labels to apply to this endpoint. | | map | ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | ------------- | -------------------------------------------------------------------------- | -------------------------- | -------------------------------------- | ------- | | node | The name of the node where this HostEndpoint resides. | | string | | | interfaceName | Either `*` or the name of the specific interface on which to apply policy. | | string | | | expectedIPs | The expected IP addresses associated with the interface. | Valid IPv4 or IPv6 address | list | | | profiles | The list of profiles to apply to the endpoint. | | list | | | ports | List of named ports that this workload exposes. | | List of [EndpointPorts](#endpointport) | | ### EndpointPort[​](#endpointport) An EndpointPort associates a name with a particular TCP/UDP/SCTP port of the endpoint, allowing it to be referenced as a named port in [policy rules](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy#entityrule). | Field | Description | Accepted Values | Schema | Default | | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------- | ------ | ------- | | name | The name to attach to this port, allowing it to be referred to in [policy rules](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy#entityrule). Names must be unique within an endpoint. | | string | | | protocol | The protocol of this named port. | `TCP`, `UDP`, `SCTP` | string | | | port | The workload port number. | `1`-`65535` | int | | > **SECONDARY:** On their own, EndpointPort entries don't result in any change to the connectivity of the port. They only have an effect if they are referred to in policy. ## Supported operations[​](#supported-operations) | Datastore type | Create/Delete | Update | Get/List | Notes | | --------------------- | ------------- | ------ | -------- | ----- | | Kubernetes API server | Yes | Yes | Yes | | ### IP pool An IP pool resource (`IPPool`) represents a collection of IP addresses from which Calico Cloud expects endpoint IPs to be assigned. For `kubectl` commands, the following case-insensitive aliases may be used to specify the resource type on the CLI: `ippool.projectcalico.org`, `ippools.projectcalico.org` as well as abbreviations such as `ippool.p` and `ippools.p`. ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: my.ippool-1 spec: cidr: 10.1.0.0/16 ipipMode: CrossSubnet natOutgoing: true disabled: false nodeSelector: all() allowedUses: - Workload - Tunnel ``` ## IP pool definition[​](#ip-pool-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | | ----- | ------------------------------------------- | --------------------------------------------------- | ------ | | name | The name of this IPPool resource. Required. | Alphanumeric string with optional `.`, `_`, or `-`. | string | ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- | --------------------------------------------- | | cidr | IP range to use for this pool. | A valid IPv4 or IPv6 CIDR. Subnet length must be at least big enough to fit a single block (by default `/26` for IPv4 or `/122` for IPv6). Must not overlap with the Link Local range `169.254.0.0/16` or `fe80::/10`. | string | | | blockSize | The CIDR size of allocation blocks used by this pool. Blocks are allocated on demand to hosts and are used to aggregate routes. The value can only be set when the pool is created. | 20 to 32 (inclusive) for IPv4 and 116 to 128 (inclusive) for IPv6 | int | `26` for IPv4 pools and `122` for IPv6 pools. | | ipipMode | The mode defining when IPIP will be used. Cannot be set at the same time as `vxlanMode`. | Always, CrossSubnet, Never | string | `Never` | | vxlanMode | The mode defining when VXLAN will be used. Cannot be set at the same time as `ipipMode`. | Always, CrossSubnet, Never | string | `Never` | | natOutgoing | When enabled, packets sent from Calico Cloud networked containers in this pool to destinations outside of any Calico IP pools will be masqueraded. | true, false | boolean | `false` | | disabled | When set to true, Calico Cloud IPAM will not assign addresses from this pool. | true, false | boolean | `false` | | disableBGPExport *(since v3.11.0)* | Disable exporting routes from this IP Pool’s CIDR over BGP. | true, false | boolean | `false` | | nodeSelector | Selects the nodes where Calico Cloud IPAM should assign pod addresses from this pool. Can be overridden if a pod [explicitly identifies this IP pool by annotation](https://docs.tigera.io/calico-cloud/reference/component-resources/configuration#using-kubernetes-annotations). | | [selector](#node-selector) | all() | | allowedUses *(since v3.11.0)* | Controls whether the pool will be used for automatic assignments of certain types. See [below](#allowed-uses). | Workload, Tunnel, HostSecondaryInterface, LoadBalancer | list of strings | `["Workload", "Tunnel"]` | | awsSubnetID *(since v3.11.0)* | May be set to the ID of an AWS VPC Subnet that contains the CIDR of this IP pool to activate the AWS-backed pool feature. See [below](#aws-backed-pools). | Valid AWS Subnet ID. | string | | | assignmentMode | Controls whether the pool will be used for automatic assignments or only if requested manually | Automatic, Manual | strings | `Automatic` | > **SECONDARY:** Do not use a custom `blockSize` until **all** Calico Cloud components have been updated to a version that supports it (at least v2.3.0). Older versions of components do not understand the field so they may corrupt the IP pool by creating blocks of incorrect size. ### Allowed uses[​](#allowed-uses) When automatically assigning IP addresses to workloads, only pools with "Workload" in their `allowedUses` field are consulted. Similarly, when assigning IPs for tunnel devices, only "Tunnel" pools are eligible. Finally, when assigning IP addresses for AWS secondary ENIs, only pools with allowed use "HostSecondaryInterface" are candidates. Combining options for the `allowedUses` field is limited. You can specify only the following options and option combinations: - `allowedUses: ["Tunnel","Workload"]` (default) - `allowedUses: ["Tunnel"]` - `allowedUses: ["Workload"]` - `allowedUses: ["LoadBalancer"]` - `allowedUses: ["HostSecondaryInterface"]` If the `allowedUses` field is not specified, it defaults to `["Workload", "Tunnel"]` for compatibility with older versions of Calico. It is not possible to specify a pool with no allowed uses. The `allowedUses` field is only consulted for new allocations, changing the field has no effect on previously allocated addresses. Calico Cloud supports Kubernetes [annotations that force the use of specific IP addresses](https://docs.tigera.io/calico-cloud/reference/component-resources/configuration#requesting-a-specific-ip-address). These annotations take precedence over the `allowedUses` field. ### AWS-backed pools[​](#aws-backed-pools) Calico Cloud supports IP pools that are backed by the AWS fabric. This feature was added in order to support egress gateways on the AWS fabric; the restrictions and requirements are currently documented as part of the [egress gateways on AWS guide](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-aws). ### IPIP[​](#ipip) Routing of packets using IP-in-IP will be used when the destination IP address is in an IP Pool that has IPIP enabled. In addition, if the `ipipMode` is set to `CrossSubnet`, Calico Cloud will only route using IP-in-IP if the IP address of the destination node is in a different subnet. The subnet of each node is configured on the node resource (which may be automatically determined when running the `cnx-node` service). For details on configuring IP-in-IP on your deployment, please refer to [Configuring IP-in-IP](https://docs.tigera.io/calico-cloud/networking/configuring/vxlan-ipip). > **SECONDARY:** Setting `natOutgoing` is recommended on any IP Pool with `ipip` enabled. When `ipip` is enabled without `natOutgoing` routing between Workloads and Hosts running Calico Cloud is asymmetric and may cause traffic to be filtered due to [RPF](https://en.wikipedia.org/wiki/Reverse_path_forwarding) checks failing. ### VXLAN[​](#vxlan) Routing of packets using VXLAN will be used when the destination IP address is in an IP Pool that has VXLAN enabled. In addition, if the `vxlanMode` is set to `CrossSubnet`, Calico Cloud will only route using VXLAN if the IP address of the destination node is in a different subnet. The subnet of each node is configured on the node resource (which may be automatically determined when running the `cnx-node` service). > **SECONDARY:** Setting `natOutgoing` is recommended on any IP Pool with `vxlan` enabled. When `vxlan` is enabled without `natOutgoing` routing between Workloads and Hosts running Calico Cloud is asymmetric and may cause traffic to be filtered due to [RPF](https://en.wikipedia.org/wiki/Reverse_path_forwarding) checks failing. ### Block sizes[​](#block-sizes) The default block sizes of `26` for IPv4 and `122` for IPv6 provide blocks of 64 addresses. This allows addresses to be allocated in groups to workloads running on the same host. By grouping addresses, fewer routes need to be exchanged between hosts and to other BGP peers. If a host allocates all of the addresses in a block then it will be allocated an additional block. If there are no more blocks available then the host can take addresses from blocks allocated to other hosts. Specific routes are added for the borrowed addresses which has an impact on route table size. Increasing the block size from the default (e.g., using `24` for IPv4 to give 256 addresses per block) means fewer blocks per host, and potentially fewer routes. But try to ensure that there are at least as many blocks in the pool as there are hosts. Reducing the block size from the default (e.g., using `28` for IPv4 to give 16 addresses per block) means more blocks per host and therefore potentially more routes. This can be beneficial if it allows the blocks to be more fairly distributed amongst the hosts. ### Node Selector[​](#node-selector) For details on configuring IP pool node selectors, please read the [Assign IP addresses based on topology guide.](https://docs.tigera.io/calico-cloud/networking/ipam/assign-ip-addresses-topology). #### Selector reference[​](#selector-reference) A label selector is an expression which either matches or does not match a resource based on its labels. Calico Cloud label selectors support a number of operators, which can be combined into larger expressions using the boolean operators and parentheses. | Expression | Meaning | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Logical operators** | | | `( )` | Matches if and only if `` matches. (Parentheses are used for grouping expressions.) | | `! ` | Matches if and only if `` does not match. **Tip:** `!` is a special character at the start of a YAML string, if you need to use `!` at the start of a YAML string, enclose the string in quotes. | | ` && ` | "And": matches if and only if both ``, and, `` matches | | ` \|\| ` | "Or": matches if and only if either ``, or, `` matches. | | **Match operators** | | | `all()` | Match all in-scope resources. To match *no* resources, combine this operator with `!` to form `!all()`. | | `global()` | Match all non-namespaced resources. Useful in a `namespaceSelector` to select global resources such as global network sets. | | `k == 'v'` | Matches resources with the label 'k' and value 'v'. | | `k != 'v'` | Matches resources without label 'k' or with label 'k' and value *not* equal to `v` | | `has(k)` | Matches resources with label 'k', independent of value. To match pods that do not have label `k`, combine this operator with `!` to form `!has(k)` | | `k in { 'v1', 'v2' }` | Matches resources with label 'k' and value in the given set | | `k not in { 'v1', 'v2' }` | Matches resources without label 'k' or with label 'k' and value *not* in the given set | | `k contains 's'` | Matches resources with label 'k' and value containing the substring 's' | | `k starts with 's'` | Matches resources with label 'k' and value starting with the substring 's' | | `k ends with 's'` | Matches resources with label 'k' and value ending with the substring 's' | Operators have the following precedence: - **Highest**: all the match operators - Parentheses `( ... )` - Negation with `!` - Conjunction with `&&` - **Lowest**: Disjunction with `||` For example, the expression ```text ! has(my-label) || my-label starts with 'prod' && role in {'frontend','business'} ``` Would be "bracketed" like this: ```text ((!(has(my-label)) || ((my-label starts with 'prod') && (role in {'frontend','business'})) ``` It would match: - Any resource that did not have label "my-label". - Any resource that both: - Has a value for `my-label` that starts with "prod", and, - Has a role label with value either "frontend", or "business". ## Supported operations[​](#supported-operations) | Datastore type | Create/Delete | Update | Get/List | Notes | | --------------------- | ------------- | ------ | -------- | ----- | | Kubernetes API server | Yes | Yes | Yes | | ## See also[​](#see-also) The [`IPReservation` resource](https://docs.tigera.io/calico-cloud/reference/resources/ipreservation) allows for small parts of an IP pool to be reserved so that they will not be used for automatic IPAM assignments. ### IP reservation An IP reservation resource (`IPReservation`) represents a collection of IP addresses that Calico Cloud should not use when automatically assigning new IP addresses. It only applies when Calico Cloud IPAM is in use. ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: IPReservation metadata: name: my-ipreservation-1 spec: reservedCIDRs: - 192.168.2.3 - 10.0.2.3/32 - cafe:f00d::/123 ``` ## IP reservation definition[​](#ip-reservation-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | | ----- | -------------------------------------------------- | --------------------------------------------------- | ------ | | name | The name of this IPReservation resource. Required. | Alphanumeric string with optional `.`, `_`, or `-`. | string | ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | ------------- | --------------------------------------------------------------- | -------------------------------------------------- | ------ | ------- | | reservedCIDRs | List of IP addresses and/or networks specified in CIDR notation | List of valid IP addresses (v4 or v6) and/or CIDRs | list | | ### Notes[​](#notes) The implementation of `IPReservation`s is designed to handle reservation of a small number of IP addresses/CIDRs from (generally much larger) IP pools. If a significant portion of an IP pool is reserved (say more than 10%) then Calico Cloud may become significantly slower when searching for free IPAM blocks. Since `IPReservations` must be consulted for every IPAM assignment request, it's best to have one or two `IPReservation` resources with multiple addresses per `IPReservation` resource (rather than having many IPReservation resources), each with one address inside. If an `IPReservation` is created after an IP from its range is already in use then the IP is not automatically released back to the pool. The reservation check is only done at auto allocation time. Calico Cloud supports Kubernetes [annotations that force the use of specific IP addresses](https://docs.tigera.io/calico-cloud/reference/component-resources/configuration#requesting-a-specific-ip-address). These annotations override any `IPReservation`s that are in place. When Windows nodes claim blocks of IPs they automatically assign the first three IPs in each block and the final IP for internal purposes. These assignments cannot be blocked by an `IPReservation`. However, if a whole IPAM block is reserved with an `IPReservation`, Windows nodes will not claim such a block. ### IPAM configuration An IPAM configuration resource (`IPAMConfiguration`) represents global IPAM configuration options. ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: IPAMConfiguration metadata: name: default spec: strictAffinity: false maxBlocksPerHost: 4 ``` ## IPAM configuration definition[​](#ipam-configuration-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | | ----- | --------------------------------------------------------- | --------------- | ------ | | name | Unique name to describe this resource instance. Required. | default | string | The resource is a singleton which must have the name `default`. ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | ---------------- | ------------------------------------------------------------------- | --------------- | ------ | ------- | | strictAffinity | When StrictAffinity is true, borrowing IP addresses is not allowed. | true, false | bool | false | | maxBlocksPerHost | The max number of blocks that can be affine to each host. | 0 - max(int32) | int | 20 | ## Supported operations[​](#supported-operations) | Datastore type | Create | Delete | Update | Get/List | | --------------------- | ------ | ------ | ------ | -------- | | etcdv3 | Yes | Yes | Yes | Yes | | Kubernetes API server | Yes | Yes | Yes | Yes | ### License key A License Key resource (`LicenseKey`) represents a user's license to use Calico Cloud. Keys are provided by Tigera support, and must be applied to the cluster to enable Calico Cloud features. For `kubectl` commands, the following case-insensitive aliases may be used to specify the resource type on the CLI: `licensekey.projectcalico.org`, `licensekeys.projectcalico.org` as well as abbreviations such as `licensekey.p` and `licensekeys.p`. ## Working with license keys[​](#working-with-license-keys) ### Applying or updating a license key[​](#applying-or-updating-a-license-key) When you add Calico Cloud to an existing Kubernetes cluster or create a new OpenShift cluster, you must apply your license key to complete the installation and gain access to the full set of Calico Cloud features. When your license key expires, you must update it to continue using Calico Cloud. To apply or update a license key use the following command, replacing `` with the customer name in the file sent to you by Tigera. **Command** ```bash kubectl apply -f -license.yaml ``` **Example** ```bash kubectl apply -f awesome-corp-license.yaml ``` ### Viewing information about your license key[​](#viewing-information-about-your-license-key) To view the number of licensed nodes and the license key expiry, use: ```bash kubectl get licensekeys.p -o custom-columns='Name:.metadata.name,MaxNodes:.status.maxnodes,Expiry:.status.expiry,PackageType:.status.package' ``` This is an example of the output of above command. ```text Name MaxNodes Expiry Package default 100 2021-10-01T23:59:59Z Enterprise ``` ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: LicenseKey metadata: creationTimestamp: null name: default spec: certificate: | -----BEGIN CERTIFICATE----- MII...n5 -----END CERTIFICATE----- token: eyJ...zaQ status: expiry: '2021-10-01T23:59:59Z' maxnodes: 100 package: Enterprise ``` The data fields in the license key resource may change without warning. The license key resource is currently a singleton: the only valid name is `default`. ## Supported operations[​](#supported-operations) | Datastore type | Create | Delete | Update | Get/List | Notes | | --------------------- | ------ | ------ | ------ | -------- | ----- | | Kubernetes API server | Yes | No | Yes | Yes | | ### Kubernetes controllers configuration A Calico Cloud [Kubernetes controllers](https://docs.tigera.io/calico-cloud/reference/component-resources/kube-controllers/configuration) configuration resource (`KubeControllersConfiguration`) represents configuration options for the Calico Cloud Kubernetes controllers. ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: KubeControllersConfiguration metadata: name: default spec: logSeverityScreen: Info healthChecks: Enabled prometheusMetricsPort: 9094 controllers: node: reconcilerPeriod: 5m leakGracePeriod: 15m syncLabels: Enabled hostEndpoint: autoCreate: Disabled createDefaultHostEndpoint: Enabled templates: - generateName: custom-host-endpoint interfaceCIDRs: - 1.2.3.0/24 interfacePattern: "eth0|eth1" nodeSelector: "has(my-label)" labels: key: value loadbalancer: assignIPs: AllServices ``` ## Kubernetes controllers configuration definition[​](#kubernetes-controllers-configuration-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | | ----- | --------------------------------------------------------- | ----------------- | ------ | | name | Unique name to describe this resource instance. Required. | Must be `default` | string | - Calico Cloud automatically creates a resource named `default` containing the configuration settings, only the name `default` is used and only one object of this type is allowed. ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | --------------------- | --------------------------------------------------------- | ----------------------------------- | --------------------------- | ------- | | logSeverityScreen | The log severity above which logs are sent to the stdout. | Debug, Info, Warning, Error, Fatal | string | Info | | healthChecks | Enable support for health checks | Enabled, Disabled | string | Enabled | | prometheusMetricsPort | Port on which to serve prometheus metrics. | Set to 0 to disable, > 0 to enable. | TCP port | 9094 | | controllers | Enabled controllers and their settings | | [Controllers](#controllers) | | ### Controllers[​](#controllers) | Field | Description | Schema | | ----------------- | ------------------------------------------------------ | ------------------------------------------------------------------------------- | | node | Enable and configure the node controller | omit to disable, or [NodeController](#nodecontroller) | | federatedservices | Enable and configure the federated services controller | omit to disable, or [FederatedServicesController](#federatedservicescontroller) | ### NodeController[​](#nodecontroller) The node controller automatically cleans up configuration for nodes that no longer exist. Optionally, it can create host endpoints for all Kubernetes nodes. | Field | Description | Accepted Values | Schema | Default | | ---------------- | --------------------------------------------------------------------------------- | ----------------- | ------------------------------------------------------------- | ------- | | reconcilerPeriod | Period to perform reconciliation with the Calico Cloud datastore | | [Duration string](https://golang.org/pkg/time/#ParseDuration) | 5m | | syncLabels | When enabled, Kubernetes node labels will be copied to Calico Cloud node objects. | Enabled, Disabled | string | Enabled | | hostEndpoint | Configures the host endpoint controller | | [HostEndpoint](#hostendpoint) | | | leakGracePeriod | Grace period to use when garbage collecting suspected leaked IP addresses. | | [Duration string](https://golang.org/pkg/time/#ParseDuration) | 15m | ### HostEndpoint[​](#hostendpoint) | Field | Description | Accepted Values | Schema | Default | | ------------------------- | --------------------------------------------------- | ----------------- | --------------------- | -------- | | autoCreate | When enabled, automatically create host endpoints | Enabled, Disabled | string | Disabled | | createDefaultHostEndpoint | When enabled, default host endpoint will be created | Enabled, Disabled | string | Enabled | | templates | Controls creation of custom host endpoints | | [Template](#template) | | ### Template[​](#template) | Field | Description | Accepted Values | Schema | Default | | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ---------------------------------- | ------- | | generateName | Unique name used as suffix for host endpoints created based on this template | Alphanumeric string | string | | | nodeSelector | Selects the nodes for which this template should create host endpoints | | [Selector](#selectors) | all() | | interfaceCIDRs | This configuration defines which IP addresses from a node's specification (including standard, tunnel, and WireGuard IPs) are eligible for inclusion in the generated HostEndpoint. IP addresses must fall within the provided CIDR ranges to be considered. If no address on the node matches the specified CIDRs, the HostEndpoint creation is skipped. | List of valid CIDRs | List string | | | interfacePattern | Regex to include matching interfaces and their IPs | string | string | | | labels | Labels to be added to generated host endpoints matching this template | | map of string key to string values | | ### Selectors[​](#selectors) A label selector is an expression which either matches or does not match a resource based on its labels. Calico Cloud label selectors support a number of operators, which can be combined into larger expressions using the boolean operators and parentheses. | Expression | Meaning | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Logical operators** | | | `( )` | Matches if and only if `` matches. (Parentheses are used for grouping expressions.) | | `! ` | Matches if and only if `` does not match. **Tip:** `!` is a special character at the start of a YAML string, if you need to use `!` at the start of a YAML string, enclose the string in quotes. | | ` && ` | "And": matches if and only if both ``, and, `` matches | | ` \|\| ` | "Or": matches if and only if either ``, or, `` matches. | | **Match operators** | | | `all()` | Match all in-scope resources. To match *no* resources, combine this operator with `!` to form `!all()`. | | `global()` | Match all non-namespaced resources. Useful in a `namespaceSelector` to select global resources such as global network sets. | | `k == 'v'` | Matches resources with the label 'k' and value 'v'. | | `k != 'v'` | Matches resources without label 'k' or with label 'k' and value *not* equal to `v` | | `has(k)` | Matches resources with label 'k', independent of value. To match pods that do not have label `k`, combine this operator with `!` to form `!has(k)` | | `k in { 'v1', 'v2' }` | Matches resources with label 'k' and value in the given set | | `k not in { 'v1', 'v2' }` | Matches resources without label 'k' or with label 'k' and value *not* in the given set | | `k contains 's'` | Matches resources with label 'k' and value containing the substring 's' | | `k starts with 's'` | Matches resources with label 'k' and value starting with the substring 's' | | `k ends with 's'` | Matches resources with label 'k' and value ending with the substring 's' | Operators have the following precedence: - **Highest**: all the match operators - Parentheses `( ... )` - Negation with `!` - Conjunction with `&&` - **Lowest**: Disjunction with `||` For example, the expression ```text ! has(my-label) || my-label starts with 'prod' && role in {'frontend','business'} ``` Would be "bracketed" like this: ```text ((!(has(my-label)) || ((my-label starts with 'prod') && (role in {'frontend','business'})) ``` It would match: - Any resource that did not have label "my-label". - Any resource that both: - Has a value for `my-label` that starts with "prod", and, - Has a role label with value either "frontend", or "business". ### FederatedServicesController[​](#federatedservicescontroller) The federated services controller syncs Kubernetes services from remote clusters defined through [RemoteClusterConfigurations](https://docs.tigera.io/calico-cloud/reference/resources/remoteclusterconfiguration). | Field | Description | Schema | Default | | ---------------- | ---------------------------------------------------------------- | ------------------------------------------------------------- | ------- | | reconcilerPeriod | Period to perform reconciliation with the Calico Cloud datastore | [Duration string](https://golang.org/pkg/time/#ParseDuration) | 5m | ### LoadBalancerController[​](#loadbalancercontroller) The load balancer controller manages IPAM for Services of type LoadBalancer. | Field | Description | Accepted Values | Schema | Default | | --------- | ---------------------------------------------- | ---------------------------------- | ------ | ----------- | | assignIPs | Mode in which LoadBalancer controller operates | AllServices, RequestedServicesOnly | String | AllServices | ## Supported operations[​](#supported-operations) | Datastore type | Create | Delete (Global `default`) | Update | Get/List | Notes | | --------------------- | ------ | ------------------------- | ------ | -------- | ----- | | Kubernetes API server | Yes | Yes | Yes | Yes | | ### Managed Cluster A Managed Cluster resource (`ManagedCluster`) represents a cluster managed by a centralized management plane with a shared Elasticsearch. The management plane provides central control of the managed cluster and stores its logs. Calico Cloud supports connecting multiple Calico Cloud clusters as describe in the \[Multi-cluster management] installation guide. For `kubectl` commands, the following case-insensitive aliases may be used to specify the resource type on the CLI: `managedcluster`,`managedclusters`, `managedcluster.projectcalico.org`, `managedclusters.projectcalico.org` as well as abbreviations such as `managedcluster.p` and `managedclusters.p`. ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: ManagedCluster metadata: name: managed-cluster spec: operatorNamespace: tigera-operator ``` ## Managed cluster definition[​](#managed-cluster-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | | ----- | --------------------------------------------------------- | --------------------------------------------------- | ------ | | name | Unique name to describe this resource instance. Required. | Alphanumeric string with optional `.`, `_`, or `-`. | string | - `cluster` is a reserved name for the management plane and is considered an invalid value ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | -------------------- | ----------------------------------------------------------------------------------------------------------------- | --------------- | ------ | ------- | | installationManifest | Installation Manifest to be applied on a managed cluster infrastructure | None | string | `Empty` | | operatorNamespace | The namespace of the managed cluster's operator. This value is used in the generation of the InstallationManifest | None | string | `Empty` | - `installationManifest` field can be retrieved only once at creation time. Updates are not supported for this field. To extract the installation manifest at creation time `-o jsonpath="{.spec.installationManifest}"` parameters can be used with a `kubectl` command. ### Status[​](#status) Status represents the latest observed status of Managed cluster. The `status` is read-only for users and updated by the Calico Cloud components. | Field | Description | Schema | | ---------- | -------------------------------------------------------------------------- | -------------------------------------- | | conditions | List of condition that describe the current status of the Managed cluster. | List of ManagedClusterStatusConditions | **ManagedClusterStatusConditions** Conditions represent the latest observed set of conditions for a Managed cluster. The connection between a management plane and managed plane will be reported as following: - `Unknown` when no initial connection has been established - `True` when both planes have an established connection - `False` when neither planes have an established connection | Field | Description | Accepted Values | Schema | Default | | ------ | ------------------------------------------------------------------------- | -------------------------- | ------ | ------------------------- | | type | Type of status that is being reported | - | string | `ManagedClusterConnected` | | status | Status of the connection between a Managed cluster and management cluster | `Unknown`, `True`, `False` | string | `Unknown` | ### Network policy A network policy resource (`NetworkPolicy`) represents an ordered set of rules which are applied to a collection of endpoints that match a [label selector](#selector). `NetworkPolicy` is a namespaced resource. `NetworkPolicy` in a specific namespace only applies to [workload endpoint resources](https://docs.tigera.io/calico-cloud/reference/resources/workloadendpoint) in that namespace. Two resources are in the same namespace if the `namespace` value is set the same on both. See [global network policy resource](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) for non-namespaced network policy. `NetworkPolicy` resources can be used to define network connectivity rules between groups of Calico Cloud endpoints and host endpoints. NetworkPolicies are organized into [tiers](https://docs.tigera.io/calico-cloud/reference/resources/tier), which provide an additional layer of ordering—in particular, note that the `Pass` action skips to the next [tier](https://docs.tigera.io/calico-cloud/reference/resources/tier), to enable hierarchical security policy. For `kubectl` [commands](https://kubernetes.io/docs/reference/kubectl/overview/), the following case-insensitive aliases may be used to specify the resource type on the CLI: `networkpolicy.projectcalico.org`, `networkpolicies.projectcalico.org` and abbreviations such as `networkpolicy.p` and `networkpolicies.p`. ## Sample YAML[​](#sample-yaml) This sample policy allows TCP traffic from `frontend` endpoints to port 6379 on `database` endpoints. ```yaml apiVersion: projectcalico.org/v3 kind: NetworkPolicy metadata: name: internal-access.allow-tcp-6379 namespace: production spec: tier: internal-access selector: role == 'database' types: - Ingress - Egress ingress: - action: Allow metadata: annotations: from: frontend to: database protocol: TCP source: selector: role == 'frontend' destination: ports: - 6379 egress: - action: Allow ``` ## Definition[​](#definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | Default | | --------- | ------------------------------------------------------------------ | --------------------------------------------------- | ------ | --------- | | name | The name of the network policy. Required. | Alphanumeric string with optional `.`, `_`, or `-`. | string | | | namespace | Namespace provides an additional qualification to a resource name. | | string | "default" | ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------- | --------------------- | --------------------------------------------- | | order | Controls the order of precedence. Calico Cloud applies the policy with the lowest value first. | | float | | | tier | Name of the [tier](https://docs.tigera.io/calico-cloud/reference/resources/tier) this policy belongs to. | | string | `default` | | selector | Selects the endpoints to which this policy applies. | | [selector](#selector) | all() | | types | Applies the policy based on the direction of the traffic. To apply the policy to inbound traffic, set to `Ingress`. To apply the policy to outbound traffic, set to `Egress`. To apply the policy to both, set to `Ingress, Egress`. | `Ingress`, `Egress` | List of strings | Depends on presence of ingress/egress rules\* | | ingress | Ordered list of ingress rules applied by policy. | | List of [Rule](#rule) | | | egress | Ordered list of egress rules applied by this policy. | | List of [Rule](#rule) | | | serviceAccountSelector | Selects the service account(s) to which this policy applies. Select a specific service account by name using the `projectcalico.org/name` label. | | [selector](#selector) | all() | | performanceHints | Contains a list of hints to Calico's policy engine to help process the policy more efficiently. Hints never change the enforcement behaviour of the policy. The available hints are described [below](#performance-hints). | `AssumeNeededOnEveryNode` | List of strings | | \* If `types` has no value, Calico Cloud defaults as follows. > | Ingress Rules Present | Egress Rules Present | `Types` value | > | --------------------- | -------------------- | ----------------- | > | No | No | `Ingress` | > | Yes | No | `Ingress` | > | No | Yes | `Egress` | > | Yes | Yes | `Ingress, Egress` | ### Rule[​](#rule) A single rule matches a set of packets and applies some action to them. When multiple rules are specified, they are executed in order. | Field | Description | Accepted Values | Schema | Default | | ----------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------ | ----------------------------- | ------- | | metadata | Per-rule metadata. | | [RuleMetadata](#rulemetadata) | | | action | Action to perform when matching this rule. | `Allow`, `Deny`, `Log`, `Pass` | string | | | protocol | Positive protocol match. | `TCP`, `UDP`, `ICMP`, `ICMPv6`, `SCTP`, `UDPLite`, `1`-`255` | string \| integer | | | notProtocol | Negative protocol match. | `TCP`, `UDP`, `ICMP`, `ICMPv6`, `SCTP`, `UDPLite`, `1`-`255` | string \| integer | | | icmp | ICMP match criteria. | | [ICMP](#icmp) | | | notICMP | Negative match on ICMP. | | [ICMP](#icmp) | | | ipVersion | Positive IP version match. | `4`, `6` | integer | | | source | Source match parameters. | | [EntityRule](#entityrule) | | | destination | Destination match parameters. | | [EntityRule](#entityrule) | | | http | Match HTTP request parameters. Application layer policy must be enabled to use this field. | | [HTTPMatch](#httpmatch) | | After a `Log` action, processing continues with the next rule; `Allow` and `Deny` are immediate and final and no further rules are processed. An `action` of `Pass` in a `NetworkPolicy` or `GlobalNetworkPolicy` will skip over the remaining policies and jump to the first profile assigned to the endpoint, applying the policy configured in the profile; if there are no Profiles configured for the endpoint the default applied action is `Deny`. ### RuleMetadata[​](#rulemetadata) Metadata associated with a specific rule (rather than the policy as a whole). The contents of the metadata does not affect how a rule is interpreted or enforced; it is simply a way to store additional information for use by operators or applications that interact with Calico Cloud. | Field | Description | Schema | Default | | ----------- | ----------------------------------- | ----------------------- | ------- | | annotations | Arbitrary non-identifying metadata. | map of string to string | | Example: ```yaml metadata: annotations: app: database owner: devops ``` Annotations follow the [same rules as Kubernetes for valid syntax and character set](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set). On Linux with the iptables data plane, rule annotations are rendered as comments in the form `-m comment --comment "="` on the iptables rule(s) that correspond to the Calico Cloud rule. ### ICMP[​](#icmp) | Field | Description | Accepted Values | Schema | Default | | ----- | ------------------- | -------------------- | ------- | ------- | | type | Match on ICMP type. | Can be integer 0-254 | integer | | | code | Match on ICMP code. | Can be integer 0-255 | integer | | ### EntityRule[​](#entityrule) Entity rules specify the attributes of the source or destination of a packet that must match for the rule as a whole to match. Packets can be matched on combinations of: - Identity of the source/destination, by using [Selectors](#selectors) or by specifying a particular Kubernetes `Service`. Selectors can match [workload endpoints](https://docs.tigera.io/calico-cloud/reference/resources/workloadendpoint), [host endpoints](https://docs.tigera.io/calico-cloud/reference/resources/hostendpoint) and ([namespaced](https://docs.tigera.io/calico-cloud/reference/resources/networkset) or [global](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkset)) network sets. - Source/destination IP address, protocol and port. If the rule contains multiple match criteria (for example, an IP and a port) then all match criteria must match for the rule as a whole to match a packet. | Field | Description | Accepted Values | Schema | Default | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------------------------- | ------- | | nets | Match packets with IP in any of the listed CIDRs. | List of valid IPv4 CIDRs or list of valid IPv6 CIDRs (IPv4 and IPv6 CIDRs shouldn't be mixed in one rule) | list of cidrs | | | notNets | Negative match on CIDRs. Match packets with IP not in any of the listed CIDRs. | List of valid IPv4 CIDRs or list of valid IPv6 CIDRs (IPv4 and IPv6 CIDRs shouldn't be mixed in one rule) | list of cidrs | | | selector | Positive match on selected endpoints. If a `namespaceSelector` is also defined, the set of endpoints this applies to is limited to the endpoints in the selected namespaces. | Valid selector | [selector](#selector) | | | notSelector | Negative match on selected endpoints. If a `namespaceSelector` is also defined, the set of endpoints this applies to is limited to the endpoints in the selected namespaces. | Valid selector | [selector](#selector) | | | namespaceSelector | Positive match on selected namespaces. If specified, only workload endpoints in the selected Kubernetes namespaces are matched. Matches namespaces based on the labels that have been applied to the namespaces. Defines the scope that selectors will apply to, if not defined then selectors apply to the NetworkPolicy's namespace. Match a specific namespace by name using the `projectcalico.org/name` label. Select the non-namespaced resources like GlobalNetworkSet(s), host endpoints to which this policy applies by using `global()` selector. | Valid selector | [selector](#selector) | | | ports | Positive match on the specified ports | | list of [ports](#ports) | | | domains | Positive match on [domain names](#exact-and-wildcard-domain-names). | List of [exact or wildcard domain names](#exact-and-wildcard-domain-names) | list of strings | | | notPorts | Negative match on the specified ports | | list of [ports](#ports) | | | serviceAccounts | Match endpoints running under service accounts. If a `namespaceSelector` is also defined, the set of service accounts this applies to is limited to the service accounts in the selected namespaces. | | [ServiceAccountMatch](#serviceaccountmatch) | | | services | Match the specified service(s). If specified on egress rule destinations, no other selection criteria can be set. If specified on ingress rule sources, only positive or negative matches on ports can be specified. | | [ServiceMatch](#servicematch) | | > **SECONDARY:** You cannot mix IPv4 and IPv6 CIDRs in a single rule using `nets` or `notNets`. If you need to match both, create 2 rules. #### Selector performance in EntityRules[​](#selector-performance-in-entityrules) When rendering policy into the data plane, Calico Cloud must identify the endpoints that match the selectors in all active rules. This calculation is optimized for certain common selector types. Using the optimized selector types reduces CPU usage (and policy rendering time) by orders of magnitude. This becomes important at high scale (hundreds of active rules, hundreds of thousands of endpoints). The optimized operators are as follows: - `label == "value"` - `label in { 'v1', 'v2' }` - `has(label)` - ` && ` is optimized if **either** `` or `` is optimized. The following perform like `has(label)`. All endpoints with the label will be scanned to find matches: - `label contains 's'` - `label starts with 's'` - `label ends with 's'` The other operators, and in particular, `all()`, `!`, `||` and `!=` are not optimized. Examples: - `a == 'b'` - optimized - `a == 'b' && has(c)` - optimized - `a == 'b' || has(c)` - **not** optimized due to use of `||` - `c != 'd'` - **not** optimized due to use of `!=` - `!has(a)` - **not** optimized due to use of `!` - `a == 'b' && c != 'd'` - optimized, `a =='b'` is optimized so `a == 'b' && ` is optimized. - `c != 'd' && a == 'b'` - optimized, `a =='b'` is optimized so ` && a == 'b'` is optimized. ### Exact and wildcard domain names[​](#exact-and-wildcard-domain-names) The `domains` field is only valid for egress Allow rules. It restricts the rule to apply only to traffic to one of the specified domains. If this field is specified, the parent [Rule](#rule)'s `action` must be `Allow`, and `nets` and `selector` must both be left empty. When a configured domain name has no wildcard (`*`), it matches exactly that domain name. For example: - `microsoft.com` - `tigera.io` With a single asterisk in any part of the domain name, it matches 1 or more path components at that position. For example: - `*.google.com` matches `www.google.com` and `www.ipv6.google.com`, but not `google.com` - `www.*.com` matches `www.sun.com` and `www.apple.com`, but not `www.com` - `update.*.mycompany.com` matches `update.tools.mycompany.com`, `update.secure.suite.mycompany.com`, and so on **Not** supported are: - Multiple wildcards in the same domain, for example: `*.*.mycompany.com` - Asterisks that are not the entire component, for example: `www.g*.com` - A wildcard as the last component, for example: `www.mycompany.*` - More general wildcards, such as regular expressions > **SECONDARY:** Calico Cloud implements policy for domain names by learning the corresponding IPs from DNS, then programming rules to allow those IPs. This means that if multiple domain names A, B and C all map to the same IP, and there is domain-based policy to allow A, traffic to B and C will be allowed as well. ### Selector[​](#selector) A label selector is an expression which either matches or does not match a resource based on its labels. Calico Cloud label selectors support a number of operators, which can be combined into larger expressions using the boolean operators and parentheses. | Expression | Meaning | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Logical operators** | | | `( )` | Matches if and only if `` matches. (Parentheses are used for grouping expressions.) | | `! ` | Matches if and only if `` does not match. **Tip:** `!` is a special character at the start of a YAML string, if you need to use `!` at the start of a YAML string, enclose the string in quotes. | | ` && ` | "And": matches if and only if both ``, and, `` matches | | ` \|\| ` | "Or": matches if and only if either ``, or, `` matches. | | **Match operators** | | | `all()` | Match all in-scope resources. To match *no* resources, combine this operator with `!` to form `!all()`. | | `global()` | Match all non-namespaced resources. Useful in a `namespaceSelector` to select global resources such as global network sets. | | `k == 'v'` | Matches resources with the label 'k' and value 'v'. | | `k != 'v'` | Matches resources without label 'k' or with label 'k' and value *not* equal to `v` | | `has(k)` | Matches resources with label 'k', independent of value. To match pods that do not have label `k`, combine this operator with `!` to form `!has(k)` | | `k in { 'v1', 'v2' }` | Matches resources with label 'k' and value in the given set | | `k not in { 'v1', 'v2' }` | Matches resources without label 'k' or with label 'k' and value *not* in the given set | | `k contains 's'` | Matches resources with label 'k' and value containing the substring 's' | | `k starts with 's'` | Matches resources with label 'k' and value starting with the substring 's' | | `k ends with 's'` | Matches resources with label 'k' and value ending with the substring 's' | Operators have the following precedence: - **Highest**: all the match operators - Parentheses `( ... )` - Negation with `!` - Conjunction with `&&` - **Lowest**: Disjunction with `||` For example, the expression ```text ! has(my-label) || my-label starts with 'prod' && role in {'frontend','business'} ``` Would be "bracketed" like this: ```text ((!(has(my-label)) || ((my-label starts with 'prod') && (role in {'frontend','business'})) ``` It would match: - Any resource that did not have label "my-label". - Any resource that both: - Has a value for `my-label` that starts with "prod", and, - Has a role label with value either "frontend", or "business". Understanding scopes and the `all()` and `global()` operators: selectors have a scope of resources that they are matched against, which depends on the context in which they are used. For example: - The `nodeSelector` in an `IPPool` selects over `Node` resources. - The top-level selector in a `NetworkPolicy` selects over the workloads *in the same namespace* as the `NetworkPolicy`. - The top-level selector in a `GlobalNetworkPolicy` doesn't have the same restriction, it selects over all endpoints including namespaced `WorkloadEndpoint`s and non-namespaced `HostEndpoint`s. - The `namespaceSelector` in a `NetworkPolicy` (or `GlobalNetworkPolicy`) *rule* selects over the labels on namespaces rather than workloads. - The `namespaceSelector` determines the scope of the accompanying `selector` in the entity rule. If no `namespaceSelector` is present then the rule's `selector` matches the default scope for that type of policy. (This is the same namespace for `NetworkPolicy` and all endpoints/network sets for `GlobalNetworkPolicy`) - The `global()` operator can be used (only) in a `namespaceSelector` to change the scope of the main `selector` to include non-namespaced resources such as [GlobalNetworkSet](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkset). This allows namespaced `NetworkPolicy` resources to refer to global non-namespaced resources, which would otherwise be impossible. ### Ports[​](#ports) Calico Cloud supports the following syntaxes for expressing ports. | Syntax | Example | Description | | --------- | ---------- | ------------------------------------------------------------------- | | int | 80 | The exact (numeric) port specified | | start:end | 6040:6050 | All (numeric) ports within the range start ≤ x ≤ end | | string | named-port | A named port, as defined in the ports list of one or more endpoints | An individual numeric port may be specified as a YAML/JSON integer. A port range or named port must be represented as a string. For example, this would be a valid list of ports: ```yaml ports: [8080, '1234:5678', 'named-port'] ``` #### Named ports[​](#named-ports) Using a named port in an `EntityRule`, instead of a numeric port, gives a layer of indirection, allowing for the named port to map to different numeric values for each endpoint. For example, suppose you have multiple HTTP servers running as workloads; some exposing their HTTP port on port 80 and others on port 8080. In each workload, you could create a named port called `http-port` that maps to the correct local port. Then, in a rule, you could refer to the name `http-port` instead of writing a different rule for each type of server. > **SECONDARY:** Since each named port may refer to many endpoints (and Calico Cloud has to expand a named port into a set of endpoint/port combinations), using a named port is considerably more expensive in terms of CPU than using a simple numeric port. We recommend that they are used sparingly, only where the extra indirection is required. ### ServiceAccountMatch[​](#serviceaccountmatch) A ServiceAccountMatch matches service accounts in an EntityRule. | Field | Description | Schema | | -------- | ------------------------------- | --------------------- | | names | Match service accounts by name | list of strings | | selector | Match service accounts by label | [selector](#selector) | ### ServiceMatch[​](#servicematch) A ServiceMatch matches a service in an EntityRule. | Field | Description | Schema | | --------- | ------------------------ | ------ | | name | The service's name. | string | | namespace | The service's namespace. | string | ### Performance Hints[​](#performance-hints) Performance hints provide a way to tell Calico Cloud about the intended use of the policy so that it may process it more efficiently. Currently only one hint is defined: - `AssumeNeededOnEveryNode`: normally, Calico Cloud only calculates a policy's rules and selectors on nodes where the policy is actually in use (i.e. its selector matches a local endpoint). This saves work in most cases. The `AssumeNeededOnEveryNode` hint tells Calico Cloud to treat the policy as "in use" on *every* node. This is useful for large policy sets that are known to apply to all (or nearly all) endpoints. It effectively "preloads" the policy on every node so that there is less work to do when the first endpoint matching the policy shows up. It also prevents work from being done to tear down the policy when the last endpoint is drained. ## Supported operations[​](#supported-operations) | Datastore type | Create/Delete | Update | Get/List | Notes | | ------------------------ | ------------- | ------ | -------- | ----- | | Kubernetes API datastore | Yes | Yes | Yes | | ### Network set A network set resource (NetworkSet) represents an arbitrary set of IP subnetworks/CIDRs, allowing it to be matched by Calico Cloud policy. Network sets are useful for applying policy to traffic coming from (or going to) external, non-Calico Cloud, networks. `NetworkSet` is a namespaced resource. `NetworkSets` in a specific namespace only applies to [network policies](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) in that namespace. Two resources are in the same namespace if the `namespace` value is set the same on both. (See [GlobalNetworkSet](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkset) for non-namespaced network sets.) The metadata for each network set includes a set of labels. When Calico Cloud is calculating the set of IPs that should match a source/destination selector within a [network policy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) rule, it includes the CIDRs from any network sets that match the selector. > **SECONDARY:** Since Calico Cloud matches packets based on their source/destination IP addresses, Calico Cloud rules may not behave as expected if there is NAT between the Calico Cloud-enabled node and the networks listed in a network set. For example, in Kubernetes, incoming traffic via a service IP is typically SNATed by the kube-proxy before reaching the destination host so Calico Cloud's workload policy will see the kube-proxy's host's IP as the source instead of the real source. ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: NetworkSet metadata: name: external-database namespace: staging labels: role: db spec: nets: - 198.51.100.0/28 - 203.0.113.0/24 allowedEgressDomains: - db.com - '*.db.com' ``` ## Network set definition[​](#network-set-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | Default | | --------- | ------------------------------------------------------------------ | ------------------------------------------------- | ------ | --------- | | name | The name of this network set. Required. | Lower-case alphanumeric with optional `_` or `-`. | string | | | namespace | Namespace provides an additional qualification to a resource name. | | string | "default" | | labels | A set of labels to apply to this endpoint. | | map | | ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------- | ------ | ------- | | nets | The IP networks/CIDRs to include in the set. | Valid IPv4 or IPv6 CIDRs, for example "192.0.2.128/25" | list | | | allowedEgressDomains | The list of domain names that belong to this set and are honored in egress allow rules only. Domain names specified here only work to allow egress traffic from the cluster to external destinations. They don't work to *deny* traffic to destinations specified by domain name, or to allow ingress traffic from *sources* specified by domain name. | List of [exact or wildcard domain names](#exact-and-wildcard-domain-names) | list | | ### Exact and wildcard domain names[​](#exact-and-wildcard-domain-names) When a configured domain name has no wildcard (`*`), it matches exactly that domain name. For example: - `microsoft.com` - `tigera.io` With a single asterisk in any part of the domain name, it matches 1 or more path components at that position. For example: - `*.google.com` matches `www.google.com` and `www.ipv6.google.com`, but not `google.com` - `www.*.com` matches `www.sun.com` and `www.apple.com`, but not `www.com` - `update.*.mycompany.com` matches `update.tools.mycompany.com`, `update.secure.suite.mycompany.com`, and so on **Not** supported are: - Multiple wildcards in the same domain, for example: `*.*.mycompany.com` - Asterisks that are not the entire component, for example: `www.g*.com` - A wildcard as the last component, for example: `www.mycompany.*` - More general wildcards, such as regular expressions ### Node A node resource (`Node`) represents a node running Calico Cloud. When adding a host to a Calico Cloud cluster, a node resource needs to be created which contains the configuration for the `cnx-node` instance running on the host. When starting a `cnx-node` instance, the name supplied to the instance should match the name configured in the Node resource. By default, starting a `cnx-node` instance will automatically create a node resource using the `hostname` of the compute host. This resource is not supported in `kubectl`. ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: Node metadata: name: node-hostname spec: bgp: asNumber: 64512 ipv4Address: 10.244.0.1/24 ipv6Address: 2001:db8:85a3::8a2e:370:7334/120 ipv4IPIPTunnelAddr: 192.168.0.1 ``` ## Definition[​](#definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | | ----- | -------------------------------- | --------------------------------------------------- | ------ | | name | The name of this node. Required. | Alphanumeric string with optional `.`, `_`, or `-`. | string | ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | ---------------------------- | ------- | | bgp | BGP configuration for this node. Omit if using Calico Cloud for policy only. | | [BGP](#bgp) | | | ipv4VXLANTunnelAddr | IPv4 address of the VXLAN tunnel. This is system configured and should not be updated manually. | | string | | | ipv6VXLANTunnelAddr | IPv6 address of the VXLAN tunnel. This is system configured and should not be updated manually. | | string | | | vxlanTunnelMACAddr | MAC address of the VXLAN tunnel. This is system configured and should not be updated manually. | | string | | | vxlanTunnelMACAddrV6 | MAC address of the IPv6 VXLAN tunnel. This is system configured and should not be updated manually. | | string | | | orchRefs | Correlates this node to a node in another orchestrator. | | list of [OrchRefs](#orchref) | | | wireguard | WireGuard configuration for this node. This is applicable only if WireGuard is enabled in [Felix Configuration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig). | | [WireGuard](#wireguard) | | ### OrchRef[​](#orchref) | Field | Description | Accepted Values | Schema | Default | | ------------ | ------------------------------------------------ | --------------- | ------ | ------- | | nodeName | Name of this node according to the orchestrator. | | string | | | orchestrator | Name of the orchestrator. | k8s | string | | ### BGP[​](#bgp) | Field | Description | Accepted Values | Schema | Default | | ----------------------- | -------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------- | | asNumber | The AS Number of your `cnx-node`. | Optional. If omitted the global value is used (see [example modifying Global BGP settings](https://docs.tigera.io/calico-cloud/networking/configuring/bgp) for details about modifying the `asNumber` setting). | integer | | | ipv4Address | The IPv4 address and subnet exported as the next-hop for the Calico Cloud endpoints on the host | The IPv4 address must be specified if BGP is enabled. | string | | | ipv6Address | The IPv6 address and subnet exported as the next-hop for the Calico Cloud endpoints on the host | Optional | string | | | ipv4IPIPTunnelAddr | IPv4 address of the IP-in-IP tunnel. This is system configured and should not be updated manually. | Optional IPv4 address | string | | | routeReflectorClusterID | Enables this node as a route reflector within the given cluster | Optional IPv4 address | string | | ### WireGuard[​](#wireguard) | Field | Description | Accepted Values | Schema | Default | | -------------------- | ----------------------------------------------------------------------------------------- | --------------- | ------ | ------- | | interfaceIPv4Address | The IP address and subnet for the IPv4 WireGuard interface created by Felix on this node. | Optional | string | | | interfaceIPv6Address | The IP address and subnet for the IPv6 WireGuard interface created by Felix on this node. | Optional | string | | ## Supported operations[​](#supported-operations) | Datastore type | Create/Delete | Update | Get/List | Notes | | --------------------- | ------------- | ------ | -------- | --------------------------------------------------------- | | Kubernetes API server | No | Yes | Yes | `cnx-node` data is directly tied to the Kubernetes nodes. | ### PacketCapture A Packet Capture resource (`PacketCapture`) represents captured live traffic for debugging microservices and application interaction inside a Kubernetes cluster. Calico Cloud supports selecting one or multiple [WorkloadEndpoints resources](https://docs.tigera.io/calico-cloud/reference/resources/workloadendpoint) as described in the [Packet Capture](https://docs.tigera.io/calico-cloud/observability/packetcapture) guide. For `kubectl` [commands](https://kubernetes.io/docs/reference/kubectl/overview/), the following case-insensitive aliases may be used to specify the resource type on the CLI: `packetcapture`,`packetcaptures`, `packetcapture.projectcalico.org`, `packetcaptures.projectcalico.org` as well as abbreviations such as `packetcapture.p` and `packetcaptures.p`. ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: PacketCapture metadata: name: sample-capture namespace: sample-namespace spec: selector: k8s-app == "sample-app" filters: - protocol: TCP ports: - 80 ``` ```yaml apiVersion: projectcalico.org/v3 kind: PacketCapture metadata: name: sample-capture namespace: sample-namespace spec: selector: all() startTime: '2021-08-26T12:00:00Z' endTime: '2021-08-26T12:30:00Z' ``` ## Packet capture definition[​](#packet-capture-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | Default | | --------- | ------------------------------------------------------------------ | --------------------------------------------------- | ------ | --------- | | name | The name of the packet capture. Required. | Alphanumeric string with optional `.`, `_`, or `-`. | string | | | namespace | Namespace provides an additional qualification to a resource name. | | string | "default" | ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | --------- | ---------------------------------------------------------------------------------- | ----------------------- | ----------------------- | ------- | | selector | Selects the endpoints to which this packet capture applies. | | [selector](#selector) | | | filters | The ordered set of filters applied to traffic captured from an interface. | | [filters](#filters) | | | startTime | Defines the start time from which this PacketCapture will start capturing packets. | Date in RFC 3339 format | [startTime](#starttime) | | | endTime | Defines the end time at which this PacketCapture will stop capturing packets. | Date in RFC 3339 format | [endTime](#endtime) | | ### Selector[​](#selector) A label selector is an expression which either matches or does not match a resource based on its labels. Calico Cloud label selectors support a number of operators, which can be combined into larger expressions using the boolean operators and parentheses. | Expression | Meaning | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Logical operators** | | | `( )` | Matches if and only if `` matches. (Parentheses are used for grouping expressions.) | | `! ` | Matches if and only if `` does not match. **Tip:** `!` is a special character at the start of a YAML string, if you need to use `!` at the start of a YAML string, enclose the string in quotes. | | ` && ` | "And": matches if and only if both ``, and, `` matches | | ` \|\| ` | "Or": matches if and only if either ``, or, `` matches. | | **Match operators** | | | `all()` | Match all in-scope resources. To match *no* resources, combine this operator with `!` to form `!all()`. | | `global()` | Match all non-namespaced resources. Useful in a `namespaceSelector` to select global resources such as global network sets. | | `k == 'v'` | Matches resources with the label 'k' and value 'v'. | | `k != 'v'` | Matches resources without label 'k' or with label 'k' and value *not* equal to `v` | | `has(k)` | Matches resources with label 'k', independent of value. To match pods that do not have label `k`, combine this operator with `!` to form `!has(k)` | | `k in { 'v1', 'v2' }` | Matches resources with label 'k' and value in the given set | | `k not in { 'v1', 'v2' }` | Matches resources without label 'k' or with label 'k' and value *not* in the given set | | `k contains 's'` | Matches resources with label 'k' and value containing the substring 's' | | `k starts with 's'` | Matches resources with label 'k' and value starting with the substring 's' | | `k ends with 's'` | Matches resources with label 'k' and value ending with the substring 's' | Operators have the following precedence: - **Highest**: all the match operators - Parentheses `( ... )` - Negation with `!` - Conjunction with `&&` - **Lowest**: Disjunction with `||` For example, the expression ```text ! has(my-label) || my-label starts with 'prod' && role in {'frontend','business'} ``` Would be "bracketed" like this: ```text ((!(has(my-label)) || ((my-label starts with 'prod') && (role in {'frontend','business'})) ``` It would match: - Any resource that did not have label "my-label". - Any resource that both: - Has a value for `my-label` that starts with "prod", and, - Has a role label with value either "frontend", or "business". ### Filters[​](#filters) | Field | Description | Accepted Values | Schema | Default | | -------- | ------------------------------------- | ------------------------------------------------------------ | ----------------- | ------- | | protocol | Positive protocol match. | `TCP`, `UDP`, `ICMP`, `ICMPv6`, `SCTP`, `UDPLite`, `1`-`255` | string \| integer | | | ports | Positive match on the specified ports | | list of ports | | Calico Cloud supports the following syntax for expressing ports. | Syntax | Example | Description | | --------- | --------- | ---------------------------------------------------- | | int | 80 | The exact (numeric) port specified | | start:end | 6040:6050 | All (numeric) ports within the range start ≤ x ≤ end | An individual numeric port may be specified as a YAML/JSON integer. A port range must be represented as a string. Named ports are not supported by `PacketCapture`. Multiple ports can be defined to filter traffic. All specified ports or port ranges concatenated using the logical operator "OR". For example, this would be a valid list of ports: ```yaml ports: [8080, '1234:5678'] ``` Multiple filter rules can be defined to filter traffic. All rules are concatenated using the logical operator "OR". For example, filtering both TCP or UDP traffic will be defined as: ```yaml filters: - protocol: TCP - protocol: UDP ``` Within a single filter rule, protocol and list of valid ports will be concatenated using the logical operator "AND". For example, filtering TCP traffic and traffic for port 80 will be defined as: ```yaml filters: - protocol: TCP ports: [80] ``` ### StartTime[​](#starttime) Defines the start time from which this PacketCapture will start capturing packets in RFC 3339 format. If omitted or the value is in the past, the capture will start immediately. If the value is changed to a future time, capture will stop immediately and restart at that time. ```yaml startTime: '2021-08-26T12:00:00Z' ``` ### EndTime[​](#endtime) Defines the end time from which this PacketCapture will stop capturing packets in RFC 3339 format. If omitted the capture will continue indefinitely. If the value is changed to the past, capture will stop immediately. ```yaml endTime: '2021-08-26T12:30:00Z' ``` ### Status[​](#status) `PacketCaptureStatus` lists the current state of a `PacketCapture` and its generated capture files. | Field | Description | | ----- | -------------------------------------------------------------------------------------------------------------------------------- | | files | It describes the location of the packet capture files that is identified via a node, its directory and the file names generated. | ### Files[​](#files) | Field | Description | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | directory | The path inside the calico-node container for the generated files. | | fileNames | The name of the generated file for a `PacketCapture` ordered alphanumerically. The active packet capture file will be identified using the following schema: `{}.pcap`. Rotated capture files name will contain an index matching the rotation timestamp. | | node | The hostname of the Kubernetes node the files are located on. | | state | Determines whether a PacketCapture is capturing traffic from any interface attached to the current node. Possible values include: Capturing, Scheduled, Finished, Error, WaitingForTraffic | ### Remote cluster configuration A remote cluster configuration resource (RemoteClusterConfiguration) represents a cluster in a federation of clusters. Each remote cluster needs a configuration to be specified to allow the local cluster to access resources on the remote cluster. The connection is one-way: the information flows only from the remote to the local cluster. To share information from the local cluster to the remote one a remote cluster configuration resource must be created on the remote cluster. A remote cluster configuration causes Typha and `calicoq` to retrieve the following resources from a remote cluster: - [Workload endpoints](https://docs.tigera.io/calico-cloud/reference/resources/workloadendpoint) - [Host endpoints](https://docs.tigera.io/calico-cloud/reference/resources/hostendpoint) When using the Kubernetes API datastore with RBAC enabled on the remote cluster, the RBAC rules must be configured to allow access to these resources. For more details on the federation feature refer to the [Overview](https://docs.tigera.io/calico-cloud/multicluster/overview). This resource is not supported in `kubectl`. ## Sample YAML[​](#sample-yaml) For a remote Kubernetes datastore cluster: ```yaml apiVersion: projectcalico.org/v3 kind: RemoteClusterConfiguration metadata: name: cluster1 spec: datastoreType: kubernetes kubeconfig: /etc/tigera-federation-remotecluster/kubeconfig-rem-cluster-1 ``` For a remote etcdv3 cluster: ```yaml apiVersion: projectcalico.org/v3 kind: RemoteClusterConfiguration metadata: name: cluster1 spec: datastoreType: etcdv3 etcdEndpoints: 'https://10.0.0.1:2379,https://10.0.0.2:2379' ``` ## RemoteClusterConfiguration Definition[​](#remoteclusterconfiguration-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | | ----- | ---------------------------------------------- | ----------------------------------------- | ------ | | name | The name of this remote cluster configuration. | Lower-case alphanumeric with optional `-` | string | ### Spec[​](#spec) | Field | Secret key | Description | Accepted Values | Schema | Default | | ------------------- | ------------- | ------------------------------------------------------------------ | --------------------- | -------------------------- | ------- | | clusterAccessSecret | | Reference to a Secret that contains connection information | | Kubernetes ObjectReference | none | | datastoreType | datastoreType | The datastore type of the remote cluster. | `etcdv3` `kubernetes` | string | none | | etcdEndpoints | etcdEndpoints | A comma separated list of etcd endpoints. | | string | none | | etcdUsername | etcdUsername | Username for RBAC. | | string | none | | etcdPassword | etcdPassword | Password for the given username. | | string | none | | etcdKeyFile | etcdKey | Path to the etcd key file. | | string | none | | etcdCertFile | etcdCert | Path to the etcd certificate file. | | string | none | | etcdCACertFile | etcdCACert | Path to the etcd CA certificate file. | | string | none | | kubeconfig | kubeconfig | Location of the `kubeconfig` file. | | string | none | | k8sAPIEndpoint | | Location of the kubernetes API server. | | string | none | | k8sKeyFile | | Location of a client key for accessing the Kubernetes API. | | string | none | | k8sCertFile | | Location of a client certificate for accessing the Kubernetes API. | | string | none | | k8sCAFile | | Location of a CA certificate. | | string | none | | k8sAPIToken | | Token to be used for accessing the Kubernetes API. | | string | none | When using the `clusterAccessSecret` field, all other fields in the RemoteClusterconfiguration resource must be empty. When the `clusterAccessSecret` reference is used, all datastore configuration will be read from the referenced Secret using the "Secret key" fields named in the above table as the data key in the Secret. The fields read from a Secret that were file path or locations in a RemoteClusterConfiguration will be expected to be the file contents when read from a Secret. All of the fields that start with `etcd` are only valid when the DatastoreType is etcdv3 and the fields that start with `k8s` or `kube` are only valid when the datastore type is kubernetes. The `kubeconfig` field and the fields that end with `File` must be accessible to Typha and `calicoq`, this does not apply when the data is coming from a Secret referenced by `clusterAccessSecret`. When the DatastoreType is `kubernetes`, the `kubeconfig` file is optional but since it can contain all of the authentication information needed to access the Kubernetes API server it is generally easier to use than setting all the individual `k8s` fields. The other kubernetes fields can be used by themselves though or to override specific kubeconfig values. ## Supported operations[​](#supported-operations) | Datastore type | Create/Delete | Update | Get/List | Notes | | --------------------- | ------------- | ------ | -------- | ----- | | etcdv3 | Yes | Yes | Yes | | | Kubernetes API server | Yes | Yes | Yes | | ### RuntimeSecurity The **RuntimeSecurity** custom resource (CR) is used to enable and configure Container Threat Detection in a Calico Cloud managed cluster. ### Resource Definition[​](#resource-definition) ```yaml apiVersion: operator.tigera.io/v1 kind: RuntimeSecurity metadata: name: default spec: detectorConfig: - id: execution-container_deployment_command disabled: true - id: discovery-enumeration_of_linux_capabilities disabled: true runtimeExceptionList: - matching: regex processInvocation: "/bin/ls*" pod: "not-evil-pod" namespace: "default" - matching: exact pod: "nginx" namespace: default - matching: regex namespace: "company-operations" ``` ## Runtime Security Definition[​](#runtime-security-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | | ------ | ------------------------------------------ | --------------- | ------ | | name | The name of the runtime security resource. | default | string | | labels | A set of labels to apply to this resource. | | map | ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | -------------------- | --------------------------------------------------------------------------------- | --------------- | --------------------------------------------- | ------- | | detectorConfig | Configuration that allows particular threat detectors to be disabled | | [DetectorConfig](#detectorconfig) | | | runtimeExceptionList | List of entries of processes that are allowed to run that won't generate an event | | [runtimeExceptionList](#runtimeexceptionlist) | Enabled | ### DetectorConfig[​](#detectorconfig) The `DetectorConfig` by default is not present but can be used to disable particular threat detectors in the Calico Cloud Managed cluster. One entry per detector | Field | Description | Accepted Values | Schema | | -------- | ----------------------------------------------------------------- | --------------- | ------- | | id | The ID of the detector this entry applies too | | string | | disabled | Boolean represents weather the detector should be disabled or not | True, False | boolean | ### RuntimeExceptionList[​](#runtimeexceptionlist) This `RuntimeExceptionList` holds a list of entries that contain a list of supported fields by which a user can negate the generation of runtime reports. | Field | Description | Accepted Values | Schema | | ----------------- | ----------------------------------------------------------------------------------------------------------- | --------------- | ------ | | matching | Whether the entries are exact matches to fields or considered a regular expression | Exact, Regex | string | | processInvocation | The exact name or regex of the process to which a user wants to negate the generation of runtime logs | | string | | pod | The exact name or regex of the pod(s) to which a user wants to negate the generation of runtime logs | | string | | namespace | The exact name or regex of the namespace(s) for which a user wants to negate the generation of runtime logs | | string | ### Security event webhook A security event webhook (`SecurityEventWebhook`) is a cluster-scoped resource that represents instances of integrations with external systems through the webhook callback mechanism. For `kubectl` [commands](https://kubernetes.io/docs/reference/kubectl/overview/), the following case-insensitive aliases can be used to specify the resource type on the CLI: `securityeventwebhook.projectcalico.org`, `securityeventwebhooks.projectcalico.org` and abbreviations such as `securityeventwebhook.p` and `securityeventwebhooks.p`. ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: SecurityEventWebhook metadata: name: jira-webhook annotations: webhooks.projectcalico.org/labels: 'Cluster name:Calico Enterprise' spec: consumer: Jira state: Enabled query: type=waf config: - name: url value: 'https://your-jira-instance-name.atlassian.net/rest/api/2/issue/' - name: project value: PRJ - name: issueType value: Bug - name: username valueFrom: secretKeyRef: name: jira-secrets key: username - name: apiToken valueFrom: secretKeyRef: name: jira-secrets key: token ``` ## Security event webhook definition[​](#security-event-webhook-definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | | ----- | --------------------------------------------------------- | --------------------------------------------------- | ------ | | name | Unique name to describe this resource instance. Required. | Alphanumeric string with optional `.`, `_`, or `-`. | string | #### Annotations[​](#annotations) Security event webhooks provide an easy way to add arbitrary data to the webhook generated HTTP payload through the metadata annotation. The value of the `webhooks.projectcalico.org/labels`, if present, will be converted into the payload labels. The value must conform to the following rules: - Key and value data for a single label are separated by the `:` character, - Multiple labels are separated by the `,` character. ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Required | | -------- | -------------------------------------------------------------------------------------------------------------- | ---------------------------------- | ----------------------------------------------------------------------- | -------- | | consumer | Specifies intended consumer of the webhook. | Slack, Jira, Alertmanager, Generic | string | yes | | state | Defines current state of the webhook. | Enabled, Disabled, Debug | string | yes | | query | Defines query used to retrieve security events from Calico. | [see Query](#query) | string | yes | | config | Webhook configuration, required contents of this structure is determined by the value of the `consumer` field. | [see Config](#configuration) | list of [SecurityEventWebhookConfigVar](#securityeventwebhookconfigvar) | yes | ### SecurityEventWebhookConfigVar[​](#securityeventwebhookconfigvar) | Field | Description | Schema | Required | | --------- | ------------------------------------------------------------------------- | --------------------------------------------------------------------------- | ----------------------------------- | | name | Configuration variable name. | string | yes | | value | Direct value for the variable. | string | yes if `valueFrom` is not specified | | valueFrom | Value defined either in a Kubernetes ConfigMap or in a Kubernetes Secret. | [SecurityEventWebhookConfigVarSource](#securityeventwebhookconfigvarsource) | yes if `value` is not specified | ### SecurityEventWebhookConfigVarSource[​](#securityeventwebhookconfigvarsource) | Field | Description | Schema | Required | | --------------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------ | ----------------------------------------- | | configMapKeyRef | Kubernetes ConfigMap reference. | `ConfigMapKeySelector` (referenced ConfigMap key should exist in the `tigera-intrusion-detection` namespace) | yes if `secretKeyRef` is not specified | | secretKeyRef | Kubernetes Secret reference. | `SecretKeySelector` (referenced Secret key should exist in the `tigera-intrusion-detection` namespace) | yes if `configMapKeyRef` is not specified | ### Status[​](#status) Field `status` reflects the health of a webhook. It is a list of [Kubernetes Conditions](https://pkg.go.dev/k8s.io/apimachinery@v0.23.0/pkg/apis/meta/v1#Condition). ## Query[​](#query) Security event webhooks use a domain-specific query language to select which records from the data set should trigger the HTTP request. The query language is composed of any number of selectors, combined with boolean expressions (`AND`, `OR`, and `NOT`), set expressions (`IN` and `NOTIN`) and bracketed subexpressions. These are translated by Calico Cloud to Elastic DSL queries that are executed on the backend. Set expressions support wildcard operators asterisk (`*`) and question mark (`?`). The asterisk sign matches zero or more characters and the question mark matches a single character. A selector consists of a key, comparator, and value. Keys and values may be identifiers consisting of alphanumerics and underscores (`_`) with the first character being alphabetic or an underscore, or may be quoted strings. Values may also be integer or floating point numbers. Comparators may be `=` (equal), `!=` (not equal), `<` (less than), `<=` (less than or equal), `>` (greater than), or `>=` (greater than or equal). ## Configuration[​](#configuration) Data required to be present in the `config` section of the security event webhook `spec` depends on the intended consumer for the HTTP requests generated by the webhook. The value in the `consumer` field of the `spec` specifies the consumer and therefore data that is required to be present. Currently Calico supports the following consumers: `Slack`, `Jira`, `Alertmanager` and `Generic`. Payloads generated by the webhook will be different for each of the listed use cases. ### Slack[​](#slack) Data fields required for the `Slack` value present in the `spec.consumer` field of a webhook: | Field | Description | Required | | ----- | ------------------------------------------------------------------------------- | -------- | | url | A valid Slack [Incoming Webhook URL](https://api.slack.com/messaging/webhooks). | yes | ### Generic[​](#generic) Data fields required for the `Generic` value present in the `spec.consumer` field of a webhook: | Field | Description | Required | | ----- | ---------------------------------------------------- | -------- | | url | A generic and valid URL of another HTTP(s) endpoint. | yes | ### Jira[​](#jira) Data fields required for the `Jira` value present in the `spec.consumer` field of a webhook: | Field | Description | Required | | --------- | ---------------------------------------------------------------------- | -------- | | url | URL of a Jira REST API v2 endpoint for the organisation. | yes | | project | A valid Jira project abbreviation. | yes | | issueType | A valid issue type for the selected project, examples: `Bug` or `Task` | yes | | username | A valid Jira user name. | yes | | apiToken | A valid Jira API token for the user. | yes | ### Alertmanager[​](#alertmanager) Data fields required for the `Alertmanager` value present in the `spec.consumer` field of a webhook: | Field | Description | Required | | --------- | --------------------------------------------------------------------------------------- | -------- | | url | URL of the Alertmanager REST API v2 endpoint for alerts (ending with `/api/v2/alerts`). | yes | | basicAuth | MD5 checksum of username and password separated by the colon character. | no | | ca.crt | Certificate authority in PEM format (required for mTLS configuration). | no | | tls.key | Private key in PEM format (required for mTLS configuration). | no | | tls.crt | Certificate in PEM format (required for mTLS configuration). | no | ### Staged global network policy A staged global network policy resource (`StagedGlobalNetworkPolicy`) represents an ordered set of rules which are applied to a collection of endpoints that match a [label selector](#selector). These rules are used to preview network behavior and do not enforce network traffic. For enforcing network traffic, see [global network policy resource](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy). `StagedGlobalNetworkPolicy` is not a namespaced resource. `StagedGlobalNetworkPolicy` applies to [workload endpoint resources](https://docs.tigera.io/calico-cloud/reference/resources/workloadendpoint) in all namespaces, and to [host endpoint resources](https://docs.tigera.io/calico-cloud/reference/resources/hostendpoint). Select a namespace in a `StagedGlobalNetworkPolicy` in the standard selector by using `projectcalico.org/namespace` as the label name and a `namespace` name as the value to compare against, e.g., `projectcalico.org/namespace == "default"`. See [staged network policy resource](https://docs.tigera.io/calico-cloud/reference/resources/stagednetworkpolicy) for staged namespaced network policy. `StagedGlobalNetworkPolicy` resources can be used to define network connectivity rules between groups of Calico Cloud endpoints and host endpoints. StagedGlobalNetworkPolicies are organized into [tiers](https://docs.tigera.io/calico-cloud/reference/resources/tier), which provide an additional layer of ordering—in particular, note that the `Pass` action skips to the next [tier](https://docs.tigera.io/calico-cloud/reference/resources/tier), to enable hierarchical security policy. For `kubectl` [commands](https://kubernetes.io/docs/reference/kubectl/overview/), the following case-insensitive aliases may be used to specify the resource type on the CLI: `stagedglobalnetworkpolicy.projectcalico.org`, `stagedglobalnetworkpolicies.projectcalico.org` and abbreviations such as `stagedglobalnetworkpolicy.p` and `stagedglobalnetworkpolicies.p`. ## Sample YAML[​](#sample-yaml) This sample policy allows TCP traffic from `frontend` endpoints to port 6379 on `database` endpoints. ```yaml apiVersion: projectcalico.org/v3 kind: StagedGlobalNetworkPolicy metadata: name: internal-access.allow-tcp-6379 spec: tier: internal-access selector: role == 'database' types: - Ingress - Egress ingress: - action: Allow protocol: TCP source: selector: role == 'frontend' destination: ports: - 6379 egress: - action: Allow ``` ## Definition[​](#definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | Default | | ----- | ----------------------------------------- | --------------------------------------------------- | ------ | ------- | | name | The name of the network policy. Required. | Alphanumeric string with optional `.`, `_`, or `-`. | string | | ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------- | --------------------- | --------------------------------------------- | | order | Controls the order of precedence. Calico Cloud applies the policy with the lowest value first. | | float | | | tier | Name of the [tier](https://docs.tigera.io/calico-cloud/reference/resources/tier) this policy belongs to. | | string | `default` | | selector | Selects the endpoints to which this policy applies. | | [selector](#selector) | all() | | serviceAccountSelector | Selects the service account(s) to which this policy applies. Select all service accounts in the cluster with a specific name using the `projectcalico.org/name` label. | | [selector](#selector) | all() | | namespaceSelector | Selects the namespace(s) to which this policy applies. Select a specific namespace by name using the `projectcalico.org/name` label. | | [selector](#selector) | all() | | types | Applies the policy based on the direction of the traffic. To apply the policy to inbound traffic, set to `Ingress`. To apply the policy to outbound traffic, set to `Egress`. To apply the policy to both, set to `Ingress, Egress`. | `Ingress`, `Egress` | List of strings | Depends on presence of ingress/egress rules\* | | ingress | Ordered list of ingress rules applied by policy. | | List of [Rule](#rule) | | | egress | Ordered list of egress rules applied by this policy. | | List of [Rule](#rule) | | | doNotTrack\*\* | Indicates to apply the rules in this policy before any data plane connection tracking, and that packets allowed by these rules should not be tracked. | true, false | boolean | false | | preDNAT\*\* | Indicates to apply the rules in this policy before any DNAT. | true, false | boolean | false | | applyOnForward\*\* | Indicates to apply the rules in this policy on forwarded traffic as well as to locally terminated traffic. | true, false | boolean | false | | performanceHints | Contains a list of hints to Calico's policy engine to help process the policy more efficiently. Hints never change the enforcement behaviour of the policy. The available hints are described [below](#performance-hints). | `AssumeNeededOnEveryNode` | List of strings | | \* If `types` has no value, Calico Cloud defaults as follows. > | Ingress Rules Present | Egress Rules Present | `Types` value | > | --------------------- | -------------------- | ----------------- | > | No | No | `Ingress` | > | Yes | No | `Ingress` | > | No | Yes | `Egress` | > | Yes | Yes | `Ingress, Egress` | \*\* The `doNotTrack` and `preDNAT` and `applyOnForward` fields are meaningful only when applying policy to a [host endpoint](https://docs.tigera.io/calico-cloud/reference/resources/hostendpoint). Only one of `doNotTrack` and `preDNAT` may be set to `true` (in a given policy). If they are both `false`, or when applying the policy to a [workload endpoint](https://docs.tigera.io/calico-cloud/reference/resources/workloadendpoint), the policy is enforced after connection tracking and any DNAT. `applyOnForward` must be set to `true` if either `doNotTrack` or `preDNAT` is `true` because for a given policy, any untracked rules or rules before DNAT will in practice apply to forwarded traffic. See [Using Calico Cloud to Secure Host Interfaces](https://docs.tigera.io/calico-cloud/reference/host-endpoints/) for how `doNotTrack` and `preDNAT` and `applyOnForward` can be useful for host endpoints. ### Rule[​](#rule) A single rule matches a set of packets and applies some action to them. When multiple rules are specified, they are executed in order. | Field | Description | Accepted Values | Schema | Default | | ----------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------ | ----------------------------- | ------- | | metadata | Per-rule metadata. | | [RuleMetadata](#rulemetadata) | | | action | Action to perform when matching this rule. | `Allow`, `Deny`, `Log`, `Pass` | string | | | protocol | Positive protocol match. | `TCP`, `UDP`, `ICMP`, `ICMPv6`, `SCTP`, `UDPLite`, `1`-`255` | string \| integer | | | notProtocol | Negative protocol match. | `TCP`, `UDP`, `ICMP`, `ICMPv6`, `SCTP`, `UDPLite`, `1`-`255` | string \| integer | | | icmp | ICMP match criteria. | | [ICMP](#icmp) | | | notICMP | Negative match on ICMP. | | [ICMP](#icmp) | | | ipVersion | Positive IP version match. | `4`, `6` | integer | | | source | Source match parameters. | | [EntityRule](#entityrule) | | | destination | Destination match parameters. | | [EntityRule](#entityrule) | | | http | Match HTTP request parameters. Application layer policy must be enabled to use this field. | | [HTTPMatch](#httpmatch) | | After a `Log` action, processing continues with the next rule; `Allow` and `Deny` are immediate and final and no further rules are processed. An `action` of `Pass` in a `NetworkPolicy` or `GlobalNetworkPolicy` will skip over the remaining policies and jump to the first profile assigned to the endpoint, applying the policy configured in the profile; if there are no Profiles configured for the endpoint the default applied action is `Deny`. ### RuleMetadata[​](#rulemetadata) Metadata associated with a specific rule (rather than the policy as a whole). The contents of the metadata does not affect how a rule is interpreted or enforced; it is simply a way to store additional information for use by operators or applications that interact with Calico Cloud. | Field | Description | Schema | Default | | ----------- | ----------------------------------- | ----------------------- | ------- | | annotations | Arbitrary non-identifying metadata. | map of string to string | | Example: ```yaml metadata: annotations: app: database owner: devops ``` Annotations follow the [same rules as Kubernetes for valid syntax and character set](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set). On Linux with the iptables data plane, rule annotations are rendered as comments in the form `-m comment --comment "="` on the iptables rule(s) that correspond to the Calico Cloud rule. ### ICMP[​](#icmp) | Field | Description | Accepted Values | Schema | Default | | ----- | ------------------- | -------------------- | ------- | ------- | | type | Match on ICMP type. | Can be integer 0-254 | integer | | | code | Match on ICMP code. | Can be integer 0-255 | integer | | ### EntityRule[​](#entityrule) Entity rules specify the attributes of the source or destination of a packet that must match for the rule as a whole to match. Packets can be matched on combinations of: - Identity of the source/destination, by using [Selectors](#selectors) or by specifying a particular Kubernetes `Service`. Selectors can match [workload endpoints](https://docs.tigera.io/calico-cloud/reference/resources/workloadendpoint), [host endpoints](https://docs.tigera.io/calico-cloud/reference/resources/hostendpoint) and ([namespaced](https://docs.tigera.io/calico-cloud/reference/resources/networkset) or [global](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkset)) network sets. - Source/destination IP address, protocol and port. If the rule contains multiple match criteria (for example, an IP and a port) then all match criteria must match for the rule as a whole to match a packet. | Field | Description | Accepted Values | Schema | Default | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------------------------- | ------- | | nets | Match packets with IP in any of the listed CIDRs. | List of valid IPv4 CIDRs or list of valid IPv6 CIDRs (IPv4 and IPv6 CIDRs shouldn't be mixed in one rule) | list of cidrs | | | notNets | Negative match on CIDRs. Match packets with IP not in any of the listed CIDRs. | List of valid IPv4 CIDRs or list of valid IPv6 CIDRs (IPv4 and IPv6 CIDRs shouldn't be mixed in one rule) | list of cidrs | | | selector | Positive match on selected endpoints. If a `namespaceSelector` is also defined, the set of endpoints this applies to is limited to the endpoints in the selected namespaces. | Valid selector | [selector](#selector) | | | notSelector | Negative match on selected endpoints. If a `namespaceSelector` is also defined, the set of endpoints this applies to is limited to the endpoints in the selected namespaces. | Valid selector | [selector](#selector) | | | namespaceSelector | Positive match on selected namespaces. If specified, only workload endpoints in the selected Kubernetes namespaces are matched. Matches namespaces based on the labels that have been applied to the namespaces. Defines the scope that selectors will apply to, if not defined then selectors apply to the NetworkPolicy's namespace. Match a specific namespace by name using the `projectcalico.org/name` label. Select the non-namespaced resources like GlobalNetworkSet(s), host endpoints to which this policy applies by using `global()` selector. | Valid selector | [selector](#selector) | | | ports | Positive match on the specified ports | | list of [ports](#ports) | | | domains | Positive match on [domain names](#exact-and-wildcard-domain-names). | List of [exact or wildcard domain names](#exact-and-wildcard-domain-names) | list of strings | | | notPorts | Negative match on the specified ports | | list of [ports](#ports) | | | serviceAccounts | Match endpoints running under service accounts. If a `namespaceSelector` is also defined, the set of service accounts this applies to is limited to the service accounts in the selected namespaces. | | [ServiceAccountMatch](#serviceaccountmatch) | | | services | Match the specified service(s). If specified on egress rule destinations, no other selection criteria can be set. If specified on ingress rule sources, only positive or negative matches on ports can be specified. | | [ServiceMatch](#servicematch) | | > **SECONDARY:** You cannot mix IPv4 and IPv6 CIDRs in a single rule using `nets` or `notNets`. If you need to match both, create 2 rules. #### Selector performance in EntityRules[​](#selector-performance-in-entityrules) When rendering policy into the data plane, Calico Cloud must identify the endpoints that match the selectors in all active rules. This calculation is optimized for certain common selector types. Using the optimized selector types reduces CPU usage (and policy rendering time) by orders of magnitude. This becomes important at high scale (hundreds of active rules, hundreds of thousands of endpoints). The optimized operators are as follows: - `label == "value"` - `label in { 'v1', 'v2' }` - `has(label)` - ` && ` is optimized if **either** `` or `` is optimized. The following perform like `has(label)`. All endpoints with the label will be scanned to find matches: - `label contains 's'` - `label starts with 's'` - `label ends with 's'` The other operators, and in particular, `all()`, `!`, `||` and `!=` are not optimized. Examples: - `a == 'b'` - optimized - `a == 'b' && has(c)` - optimized - `a == 'b' || has(c)` - **not** optimized due to use of `||` - `c != 'd'` - **not** optimized due to use of `!=` - `!has(a)` - **not** optimized due to use of `!` - `a == 'b' && c != 'd'` - optimized, `a =='b'` is optimized so `a == 'b' && ` is optimized. - `c != 'd' && a == 'b'` - optimized, `a =='b'` is optimized so ` && a == 'b'` is optimized. ### Exact and wildcard domain names[​](#exact-and-wildcard-domain-names) The `domains` field is only valid for egress Allow rules. It restricts the rule to apply only to traffic to one of the specified domains. If this field is specified, the parent [Rule](#rule)'s `action` must be `Allow`, and `nets` and `selector` must both be left empty. When a configured domain name has no wildcard (`*`), it matches exactly that domain name. For example: - `microsoft.com` - `tigera.io` With a single asterisk in any part of the domain name, it matches 1 or more path components at that position. For example: - `*.google.com` matches `www.google.com` and `www.ipv6.google.com`, but not `google.com` - `www.*.com` matches `www.sun.com` and `www.apple.com`, but not `www.com` - `update.*.mycompany.com` matches `update.tools.mycompany.com`, `update.secure.suite.mycompany.com`, and so on **Not** supported are: - Multiple wildcards in the same domain, for example: `*.*.mycompany.com` - Asterisks that are not the entire component, for example: `www.g*.com` - A wildcard as the last component, for example: `www.mycompany.*` - More general wildcards, such as regular expressions > **SECONDARY:** Calico Cloud implements policy for domain names by learning the corresponding IPs from DNS, then programming rules to allow those IPs. This means that if multiple domain names A, B and C all map to the same IP, and there is domain-based policy to allow A, traffic to B and C will be allowed as well. ### Selector[​](#selector) A label selector is an expression which either matches or does not match a resource based on its labels. Calico Cloud label selectors support a number of operators, which can be combined into larger expressions using the boolean operators and parentheses. | Expression | Meaning | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Logical operators** | | | `( )` | Matches if and only if `` matches. (Parentheses are used for grouping expressions.) | | `! ` | Matches if and only if `` does not match. **Tip:** `!` is a special character at the start of a YAML string, if you need to use `!` at the start of a YAML string, enclose the string in quotes. | | ` && ` | "And": matches if and only if both ``, and, `` matches | | ` \|\| ` | "Or": matches if and only if either ``, or, `` matches. | | **Match operators** | | | `all()` | Match all in-scope resources. To match *no* resources, combine this operator with `!` to form `!all()`. | | `global()` | Match all non-namespaced resources. Useful in a `namespaceSelector` to select global resources such as global network sets. | | `k == 'v'` | Matches resources with the label 'k' and value 'v'. | | `k != 'v'` | Matches resources without label 'k' or with label 'k' and value *not* equal to `v` | | `has(k)` | Matches resources with label 'k', independent of value. To match pods that do not have label `k`, combine this operator with `!` to form `!has(k)` | | `k in { 'v1', 'v2' }` | Matches resources with label 'k' and value in the given set | | `k not in { 'v1', 'v2' }` | Matches resources without label 'k' or with label 'k' and value *not* in the given set | | `k contains 's'` | Matches resources with label 'k' and value containing the substring 's' | | `k starts with 's'` | Matches resources with label 'k' and value starting with the substring 's' | | `k ends with 's'` | Matches resources with label 'k' and value ending with the substring 's' | Operators have the following precedence: - **Highest**: all the match operators - Parentheses `( ... )` - Negation with `!` - Conjunction with `&&` - **Lowest**: Disjunction with `||` For example, the expression ```text ! has(my-label) || my-label starts with 'prod' && role in {'frontend','business'} ``` Would be "bracketed" like this: ```text ((!(has(my-label)) || ((my-label starts with 'prod') && (role in {'frontend','business'})) ``` It would match: - Any resource that did not have label "my-label". - Any resource that both: - Has a value for `my-label` that starts with "prod", and, - Has a role label with value either "frontend", or "business". ### Ports[​](#ports) Calico Cloud supports the following syntaxes for expressing ports. | Syntax | Example | Description | | --------- | ---------- | ------------------------------------------------------------------- | | int | 80 | The exact (numeric) port specified | | start:end | 6040:6050 | All (numeric) ports within the range start ≤ x ≤ end | | string | named-port | A named port, as defined in the ports list of one or more endpoints | An individual numeric port may be specified as a YAML/JSON integer. A port range or named port must be represented as a string. For example, this would be a valid list of ports: ```yaml ports: [8080, '1234:5678', 'named-port'] ``` #### Named ports[​](#named-ports) Using a named port in an `EntityRule`, instead of a numeric port, gives a layer of indirection, allowing for the named port to map to different numeric values for each endpoint. For example, suppose you have multiple HTTP servers running as workloads; some exposing their HTTP port on port 80 and others on port 8080. In each workload, you could create a named port called `http-port` that maps to the correct local port. Then, in a rule, you could refer to the name `http-port` instead of writing a different rule for each type of server. > **SECONDARY:** Since each named port may refer to many endpoints (and Calico Cloud has to expand a named port into a set of endpoint/port combinations), using a named port is considerably more expensive in terms of CPU than using a simple numeric port. We recommend that they are used sparingly, only where the extra indirection is required. ### ServiceAccountMatch[​](#serviceaccountmatch) A ServiceAccountMatch matches service accounts in an EntityRule. | Field | Description | Schema | | -------- | ------------------------------- | --------------------- | | names | Match service accounts by name | list of strings | | selector | Match service accounts by label | [selector](#selector) | ### ServiceMatch[​](#servicematch) A ServiceMatch matches a service in an EntityRule. | Field | Description | Schema | | --------- | ------------------------ | ------ | | name | The service's name. | string | | namespace | The service's namespace. | string | ### Performance Hints[​](#performance-hints) Performance hints provide a way to tell Calico Cloud about the intended use of the policy so that it may process it more efficiently. Currently only one hint is defined: - `AssumeNeededOnEveryNode`: normally, Calico Cloud only calculates a policy's rules and selectors on nodes where the policy is actually in use (i.e. its selector matches a local endpoint). This saves work in most cases. The `AssumeNeededOnEveryNode` hint tells Calico Cloud to treat the policy as "in use" on *every* node. This is useful for large policy sets that are known to apply to all (or nearly all) endpoints. It effectively "preloads" the policy on every node so that there is less work to do when the first endpoint matching the policy shows up. It also prevents work from being done to tear down the policy when the last endpoint is drained. ## Supported operations[​](#supported-operations) | Datastore type | Create/Delete | Update | Get/List | Notes | | ------------------------ | ------------- | ------ | -------- | ----- | | Kubernetes API datastore | Yes | Yes | Yes | | ### Staged Kubernetes network policy A staged kubernetes network policy resource (`StagedKubernetesNetworkPolicy`) represents a staged version of [Kubernetes network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies). This is used to preview network behavior before actually enforcing the network policy. Once persisted, this will create a Kubernetes network policy backed by a Calico Cloud [network policy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy). For `kubectl` [commands](https://kubernetes.io/docs/reference/kubectl/overview/), the following case-insensitive aliases may be used to specify the resource type on the CLI: `stagedkubernetesnetworkpolicy.projectcalico.org`, `stagedkubernetesnetworkpolicies.projectcalico.org` and abbreviations such as `stagedkubernetesnetworkpolicy.p` and `stagedkubernetesnetworkpolicies.p`. ## Sample YAML[​](#sample-yaml) Below is a sample policy created from the example policy from the [Kubernetes NetworkPolicy documentation](https://kubernetes.io/docs/concepts/services-networking/network-policies/#networkpolicy-resource). The only difference between this policy and the example Kubernetes version is that the `apiVersion` and `kind` are changed to properly specify a staged Kubernetes network policy. ```yaml apiVersion: projectcalico.org/v3 kind: StagedKubernetesNetworkPolicy metadata: name: test-network-policy namespace: default spec: podSelector: matchLabels: role: db policyTypes: - Ingress - Egress ingress: - from: - ipBlock: cidr: 172.17.0.0/16 except: - 172.17.1.0/24 - namespaceSelector: matchLabels: project: myproject - podSelector: matchLabels: role: frontend ports: - protocol: TCP port: 6379 egress: - to: - ipBlock: cidr: 10.0.0.0/24 ports: - protocol: TCP port: 5978 ``` ## Definition[​](#definition) See the [Kubernetes NetworkPolicy documentation](https://v1-21.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.21/#networkpolicyspec-v1-networking-k8s-io) for more information. ### Staged network policy A staged network policy resource (`StagedNetworkPolicy`) represents an ordered set of rules which are applied to a collection of endpoints that match a [label selector](#selector). These rules are used to preview network behavior and do not enforce network traffic. For enforcing network traffic, see [network policy resource](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy). `StagedNetworkPolicy` is a namespaced resource. `StagedNetworkPolicy` in a specific namespace only applies to [workload endpoint resources](https://docs.tigera.io/calico-cloud/reference/resources/workloadendpoint) in that namespace. Two resources are in the same namespace if the `namespace` value is set the same on both. See [staged global network policy resource](https://docs.tigera.io/calico-cloud/reference/resources/stagedglobalnetworkpolicy) for staged non-namespaced network policy. `StagedNetworkPolicy` resources can be used to define network connectivity rules between groups of Calico Cloud endpoints and host endpoints. StagedNetworkPolicies are organized into [tiers](https://docs.tigera.io/calico-cloud/reference/resources/tier), which provide an additional layer of ordering—in particular, note that the `Pass` action skips to the next [tier](https://docs.tigera.io/calico-cloud/reference/resources/tier), to enable hierarchical security policy. For `kubectl` [commands](https://kubernetes.io/docs/reference/kubectl/overview/), the following case-insensitive aliases may be used to specify the resource type on the CLI: `stagednetworkpolicy.projectcalico.org`, `stagednetworkpolicies.projectcalico.org` and abbreviations such as `stagednetworkpolicy.p` and `stagednetworkpolicies.p`. ## Sample YAML[​](#sample-yaml) This sample policy allows TCP traffic from `frontend` endpoints to port 6379 on `database` endpoints. ```yaml apiVersion: projectcalico.org/v3 kind: StagedNetworkPolicy metadata: name: internal-access.allow-tcp-6379 namespace: production spec: tier: internal-access selector: role == 'database' types: - Ingress - Egress ingress: - action: Allow protocol: TCP source: selector: role == 'frontend' destination: ports: - 6379 egress: - action: Allow ``` ## Definition[​](#definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | Default | | --------- | ------------------------------------------------------------------ | --------------------------------------------------- | ------ | --------- | | name | The name of the network policy. Required. | Alphanumeric string with optional `.`, `_`, or `-`. | string | | | namespace | Namespace provides an additional qualification to a resource name. | | string | "default" | ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------- | --------------------- | --------------------------------------------- | | order | Controls the order of precedence. Calico Cloud applies the policy with the lowest value first. | | float | | | tier | Name of the [tier](https://docs.tigera.io/calico-cloud/reference/resources/tier) this policy belongs to. | | string | `default` | | selector | Selects the endpoints to which this policy applies. | | [selector](#selector) | all() | | types | Applies the policy based on the direction of the traffic. To apply the policy to inbound traffic, set to `Ingress`. To apply the policy to outbound traffic, set to `Egress`. To apply the policy to both, set to `Ingress, Egress`. | `Ingress`, `Egress` | List of strings | Depends on presence of ingress/egress rules\* | | ingress | Ordered list of ingress rules applied by policy. | | List of [Rule](#rule) | | | egress | Ordered list of egress rules applied by this policy. | | List of [Rule](#rule) | | | serviceAccountSelector | Selects the service account(s) to which this policy applies. Select a specific service account by name using the `projectcalico.org/name` label. | | [selector](#selector) | all() | | performanceHints | Contains a list of hints to Calico's policy engine to help process the policy more efficiently. Hints never change the enforcement behaviour of the policy. The available hints are described [below](#performance-hints). | `AssumeNeededOnEveryNode` | List of strings | | \* If `types` has no value, Calico Cloud defaults as follows. > | Ingress Rules Present | Egress Rules Present | `Types` value | > | --------------------- | -------------------- | ----------------- | > | No | No | `Ingress` | > | Yes | No | `Ingress` | > | No | Yes | `Egress` | > | Yes | Yes | `Ingress, Egress` | ### Rule[​](#rule) A single rule matches a set of packets and applies some action to them. When multiple rules are specified, they are executed in order. | Field | Description | Accepted Values | Schema | Default | | ----------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------ | ----------------------------- | ------- | | metadata | Per-rule metadata. | | [RuleMetadata](#rulemetadata) | | | action | Action to perform when matching this rule. | `Allow`, `Deny`, `Log`, `Pass` | string | | | protocol | Positive protocol match. | `TCP`, `UDP`, `ICMP`, `ICMPv6`, `SCTP`, `UDPLite`, `1`-`255` | string \| integer | | | notProtocol | Negative protocol match. | `TCP`, `UDP`, `ICMP`, `ICMPv6`, `SCTP`, `UDPLite`, `1`-`255` | string \| integer | | | icmp | ICMP match criteria. | | [ICMP](#icmp) | | | notICMP | Negative match on ICMP. | | [ICMP](#icmp) | | | ipVersion | Positive IP version match. | `4`, `6` | integer | | | source | Source match parameters. | | [EntityRule](#entityrule) | | | destination | Destination match parameters. | | [EntityRule](#entityrule) | | | http | Match HTTP request parameters. Application layer policy must be enabled to use this field. | | [HTTPMatch](#httpmatch) | | After a `Log` action, processing continues with the next rule; `Allow` and `Deny` are immediate and final and no further rules are processed. An `action` of `Pass` in a `NetworkPolicy` or `GlobalNetworkPolicy` will skip over the remaining policies and jump to the first profile assigned to the endpoint, applying the policy configured in the profile; if there are no Profiles configured for the endpoint the default applied action is `Deny`. ### RuleMetadata[​](#rulemetadata) Metadata associated with a specific rule (rather than the policy as a whole). The contents of the metadata does not affect how a rule is interpreted or enforced; it is simply a way to store additional information for use by operators or applications that interact with Calico Cloud. | Field | Description | Schema | Default | | ----------- | ----------------------------------- | ----------------------- | ------- | | annotations | Arbitrary non-identifying metadata. | map of string to string | | Example: ```yaml metadata: annotations: app: database owner: devops ``` Annotations follow the [same rules as Kubernetes for valid syntax and character set](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set). On Linux with the iptables data plane, rule annotations are rendered as comments in the form `-m comment --comment "="` on the iptables rule(s) that correspond to the Calico Cloud rule. ### ICMP[​](#icmp) | Field | Description | Accepted Values | Schema | Default | | ----- | ------------------- | -------------------- | ------- | ------- | | type | Match on ICMP type. | Can be integer 0-254 | integer | | | code | Match on ICMP code. | Can be integer 0-255 | integer | | ### EntityRule[​](#entityrule) Entity rules specify the attributes of the source or destination of a packet that must match for the rule as a whole to match. Packets can be matched on combinations of: - Identity of the source/destination, by using [Selectors](#selectors) or by specifying a particular Kubernetes `Service`. Selectors can match [workload endpoints](https://docs.tigera.io/calico-cloud/reference/resources/workloadendpoint), [host endpoints](https://docs.tigera.io/calico-cloud/reference/resources/hostendpoint) and ([namespaced](https://docs.tigera.io/calico-cloud/reference/resources/networkset) or [global](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkset)) network sets. - Source/destination IP address, protocol and port. If the rule contains multiple match criteria (for example, an IP and a port) then all match criteria must match for the rule as a whole to match a packet. | Field | Description | Accepted Values | Schema | Default | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------------------------- | ------- | | nets | Match packets with IP in any of the listed CIDRs. | List of valid IPv4 CIDRs or list of valid IPv6 CIDRs (IPv4 and IPv6 CIDRs shouldn't be mixed in one rule) | list of cidrs | | | notNets | Negative match on CIDRs. Match packets with IP not in any of the listed CIDRs. | List of valid IPv4 CIDRs or list of valid IPv6 CIDRs (IPv4 and IPv6 CIDRs shouldn't be mixed in one rule) | list of cidrs | | | selector | Positive match on selected endpoints. If a `namespaceSelector` is also defined, the set of endpoints this applies to is limited to the endpoints in the selected namespaces. | Valid selector | [selector](#selector) | | | notSelector | Negative match on selected endpoints. If a `namespaceSelector` is also defined, the set of endpoints this applies to is limited to the endpoints in the selected namespaces. | Valid selector | [selector](#selector) | | | namespaceSelector | Positive match on selected namespaces. If specified, only workload endpoints in the selected Kubernetes namespaces are matched. Matches namespaces based on the labels that have been applied to the namespaces. Defines the scope that selectors will apply to, if not defined then selectors apply to the NetworkPolicy's namespace. Match a specific namespace by name using the `projectcalico.org/name` label. Select the non-namespaced resources like GlobalNetworkSet(s), host endpoints to which this policy applies by using `global()` selector. | Valid selector | [selector](#selector) | | | ports | Positive match on the specified ports | | list of [ports](#ports) | | | domains | Positive match on [domain names](#exact-and-wildcard-domain-names). | List of [exact or wildcard domain names](#exact-and-wildcard-domain-names) | list of strings | | | notPorts | Negative match on the specified ports | | list of [ports](#ports) | | | serviceAccounts | Match endpoints running under service accounts. If a `namespaceSelector` is also defined, the set of service accounts this applies to is limited to the service accounts in the selected namespaces. | | [ServiceAccountMatch](#serviceaccountmatch) | | | services | Match the specified service(s). If specified on egress rule destinations, no other selection criteria can be set. If specified on ingress rule sources, only positive or negative matches on ports can be specified. | | [ServiceMatch](#servicematch) | | > **SECONDARY:** You cannot mix IPv4 and IPv6 CIDRs in a single rule using `nets` or `notNets`. If you need to match both, create 2 rules. #### Selector performance in EntityRules[​](#selector-performance-in-entityrules) When rendering policy into the data plane, Calico Cloud must identify the endpoints that match the selectors in all active rules. This calculation is optimized for certain common selector types. Using the optimized selector types reduces CPU usage (and policy rendering time) by orders of magnitude. This becomes important at high scale (hundreds of active rules, hundreds of thousands of endpoints). The optimized operators are as follows: - `label == "value"` - `label in { 'v1', 'v2' }` - `has(label)` - ` && ` is optimized if **either** `` or `` is optimized. The following perform like `has(label)`. All endpoints with the label will be scanned to find matches: - `label contains 's'` - `label starts with 's'` - `label ends with 's'` The other operators, and in particular, `all()`, `!`, `||` and `!=` are not optimized. Examples: - `a == 'b'` - optimized - `a == 'b' && has(c)` - optimized - `a == 'b' || has(c)` - **not** optimized due to use of `||` - `c != 'd'` - **not** optimized due to use of `!=` - `!has(a)` - **not** optimized due to use of `!` - `a == 'b' && c != 'd'` - optimized, `a =='b'` is optimized so `a == 'b' && ` is optimized. - `c != 'd' && a == 'b'` - optimized, `a =='b'` is optimized so ` && a == 'b'` is optimized. ### Exact and wildcard domain names[​](#exact-and-wildcard-domain-names) The `domains` field is only valid for egress Allow rules. It restricts the rule to apply only to traffic to one of the specified domains. If this field is specified, the parent [Rule](#rule)'s `action` must be `Allow`, and `nets` and `selector` must both be left empty. When a configured domain name has no wildcard (`*`), it matches exactly that domain name. For example: - `microsoft.com` - `tigera.io` With a single asterisk in any part of the domain name, it matches 1 or more path components at that position. For example: - `*.google.com` matches `www.google.com` and `www.ipv6.google.com`, but not `google.com` - `www.*.com` matches `www.sun.com` and `www.apple.com`, but not `www.com` - `update.*.mycompany.com` matches `update.tools.mycompany.com`, `update.secure.suite.mycompany.com`, and so on **Not** supported are: - Multiple wildcards in the same domain, for example: `*.*.mycompany.com` - Asterisks that are not the entire component, for example: `www.g*.com` - A wildcard as the last component, for example: `www.mycompany.*` - More general wildcards, such as regular expressions > **SECONDARY:** Calico Cloud implements policy for domain names by learning the corresponding IPs from DNS, then programming rules to allow those IPs. This means that if multiple domain names A, B and C all map to the same IP, and there is domain-based policy to allow A, traffic to B and C will be allowed as well. ### Selector[​](#selector) A label selector is an expression which either matches or does not match a resource based on its labels. Calico Cloud label selectors support a number of operators, which can be combined into larger expressions using the boolean operators and parentheses. | Expression | Meaning | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Logical operators** | | | `( )` | Matches if and only if `` matches. (Parentheses are used for grouping expressions.) | | `! ` | Matches if and only if `` does not match. **Tip:** `!` is a special character at the start of a YAML string, if you need to use `!` at the start of a YAML string, enclose the string in quotes. | | ` && ` | "And": matches if and only if both ``, and, `` matches | | ` \|\| ` | "Or": matches if and only if either ``, or, `` matches. | | **Match operators** | | | `all()` | Match all in-scope resources. To match *no* resources, combine this operator with `!` to form `!all()`. | | `global()` | Match all non-namespaced resources. Useful in a `namespaceSelector` to select global resources such as global network sets. | | `k == 'v'` | Matches resources with the label 'k' and value 'v'. | | `k != 'v'` | Matches resources without label 'k' or with label 'k' and value *not* equal to `v` | | `has(k)` | Matches resources with label 'k', independent of value. To match pods that do not have label `k`, combine this operator with `!` to form `!has(k)` | | `k in { 'v1', 'v2' }` | Matches resources with label 'k' and value in the given set | | `k not in { 'v1', 'v2' }` | Matches resources without label 'k' or with label 'k' and value *not* in the given set | | `k contains 's'` | Matches resources with label 'k' and value containing the substring 's' | | `k starts with 's'` | Matches resources with label 'k' and value starting with the substring 's' | | `k ends with 's'` | Matches resources with label 'k' and value ending with the substring 's' | Operators have the following precedence: - **Highest**: all the match operators - Parentheses `( ... )` - Negation with `!` - Conjunction with `&&` - **Lowest**: Disjunction with `||` For example, the expression ```text ! has(my-label) || my-label starts with 'prod' && role in {'frontend','business'} ``` Would be "bracketed" like this: ```text ((!(has(my-label)) || ((my-label starts with 'prod') && (role in {'frontend','business'})) ``` It would match: - Any resource that did not have label "my-label". - Any resource that both: - Has a value for `my-label` that starts with "prod", and, - Has a role label with value either "frontend", or "business". ### Ports[​](#ports) Calico Cloud supports the following syntaxes for expressing ports. | Syntax | Example | Description | | --------- | ---------- | ------------------------------------------------------------------- | | int | 80 | The exact (numeric) port specified | | start:end | 6040:6050 | All (numeric) ports within the range start ≤ x ≤ end | | string | named-port | A named port, as defined in the ports list of one or more endpoints | An individual numeric port may be specified as a YAML/JSON integer. A port range or named port must be represented as a string. For example, this would be a valid list of ports: ```yaml ports: [8080, '1234:5678', 'named-port'] ``` #### Named ports[​](#named-ports) Using a named port in an `EntityRule`, instead of a numeric port, gives a layer of indirection, allowing for the named port to map to different numeric values for each endpoint. For example, suppose you have multiple HTTP servers running as workloads; some exposing their HTTP port on port 80 and others on port 8080. In each workload, you could create a named port called `http-port` that maps to the correct local port. Then, in a rule, you could refer to the name `http-port` instead of writing a different rule for each type of server. > **SECONDARY:** Since each named port may refer to many endpoints (and Calico Cloud has to expand a named port into a set of endpoint/port combinations), using a named port is considerably more expensive in terms of CPU than using a simple numeric port. We recommend that they are used sparingly, only where the extra indirection is required. ### ServiceAccountMatch[​](#serviceaccountmatch) A ServiceAccountMatch matches service accounts in an EntityRule. | Field | Description | Schema | | -------- | ------------------------------- | --------------------- | | names | Match service accounts by name | list of strings | | selector | Match service accounts by label | [selector](#selector) | ### ServiceMatch[​](#servicematch) A ServiceMatch matches a service in an EntityRule. | Field | Description | Schema | | --------- | ------------------------ | ------ | | name | The service's name. | string | | namespace | The service's namespace. | string | ### Performance Hints[​](#performance-hints) Performance hints provide a way to tell Calico Cloud about the intended use of the policy so that it may process it more efficiently. Currently only one hint is defined: - `AssumeNeededOnEveryNode`: normally, Calico Cloud only calculates a policy's rules and selectors on nodes where the policy is actually in use (i.e. its selector matches a local endpoint). This saves work in most cases. The `AssumeNeededOnEveryNode` hint tells Calico Cloud to treat the policy as "in use" on *every* node. This is useful for large policy sets that are known to apply to all (or nearly all) endpoints. It effectively "preloads" the policy on every node so that there is less work to do when the first endpoint matching the policy shows up. It also prevents work from being done to tear down the policy when the last endpoint is drained. ## Supported operations[​](#supported-operations) | Datastore type | Create/Delete | Update | Get/List | Notes | | ------------------------ | ------------- | ------ | -------- | ----- | | Kubernetes API datastore | Yes | Yes | Yes | | ### Tier A tier resource (`Tier`) represents an ordered collection of [NetworkPolicies](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) and/or [GlobalNetworkPolicies](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy). Tiers are used to divide these policies into groups of different priorities. These policies are ordered within a Tier: the additional hierarchy of Tiers provides more flexibility because the `Pass` `action` in a Rule jumps to the next Tier. Some example use cases for this are. - Allowing privileged users to define security policy that takes precedence over other users. - Translating hierarchies of physical firewalls directly into Calico Cloud policy. For `kubectl` [commands](https://kubernetes.io/docs/reference/kubectl/overview/), the following case-insensitive aliases may be used to specify the resource type on the CLI: `tier.projectcalico.org`, `tiers.projectcalico.org` and abbreviations such as `tier.p` and `tiers.p`. ## How policy is evaluated[​](#how-policy-is-evaluated) When a new connection is processed by Calico Cloud, each tier that contains a policy that applies to the endpoint processes the packet. Tiers are sorted by their `order` - smallest number first. Policies in each Tier are then processed in order. - If a [NetworkPolicy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) or [GlobalNetworkPolicy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) in the Tier `Allow`s or `Deny`s the packet, then evaluation is done: the packet is handled accordingly. - If a [NetworkPolicy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) or [GlobalNetworkPolicy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy) in the Tier `Pass`es the packet, the next Tier containing a Policy that applies to the endpoint processes the packet. If the Tier applies to the endpoint, but takes no action on the packet the packet is dropped. This behaviour can be changed by setting the `defaultAction` of a tier to `Pass`. If the last Tier applying to the endpoint `Pass`es the packet, that endpoint's \[Profiles] (profile.mdx) are evaluated. ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: Tier metadata: name: internal-access spec: order: 100 defaultAction: Deny ``` ## Definition[​](#definition) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | | ----- | --------------------- | --------------- | ------ | | name | The name of the tier. | | string | ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------ | --------------- | ------ | --------------------- | | order | (Optional) Indicates priority of this Tier, with lower order taking precedence. No value indicates highest order (lowest precedence) | | float | `nil` (highest order) | | defaultAction | (Optional) Indicates the default action, when this Tier applies to an endpoint, but takes not action on the packet | `Deny`, `Pass` | string | `Deny` | All Policies created by Calico Cloud orchestrator integrations are created in the default (last) Tier. ### Workload endpoint A workload endpoint resource (`WorkloadEndpoint`) represents an interface connecting a Calico Cloud networked container or VM to its host. Each endpoint may specify a set of labels and list of profiles that Calico Cloud will use to apply policy to the interface. A workload endpoint is a namespaced resource, that means a [NetworkPolicy](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy) in a specific namespace only applies to the WorkloadEndpoint in that namespace. Two resources are in the same namespace if the namespace value is set the same on both. This resource is not supported in `kubectl`. > **SECONDARY:** While `calicoctl` allows the user to fully manage Workload Endpoint resources, the lifecycle of these resources is generally handled by an orchestrator-specific plugin such as the Calico Cloud CNI plugin. In general, we recommend that you only use `calicoctl` to view this resource type. **Multiple networks** If multiple networks are enabled, workload endpoints will have additional labels which can be used in network policy selectors: - `projectcalico.org/network`: The name of the network specified in the NetworkAttachmentDefinition. - `projectcalico.org/network-namespace`: This namespace the network is in. - `projectcalico.org/network-interface`: The network interface for the workload endpoint. For more information, see the [multiple-networks how-to guide](https://docs.tigera.io/calico-cloud/networking/configuring/multiple-networks). ## Sample YAML[​](#sample-yaml) ```yaml apiVersion: projectcalico.org/v3 kind: WorkloadEndpoint metadata: name: node1-k8s-my--nginx--b1337a-eth0 namespace: default labels: app: frontend projectcalico.org/namespace: default projectcalico.org/orchestrator: k8s spec: node: node1 orchestrator: k8s endpoint: eth0 containerID: 1337495556942031415926535 pod: my-nginx-b1337a endpoint: eth0 interfaceName: cali0ef24ba mac: ca:fe:1d:52:bb:e9 ipNetworks: - 192.168.0.0/32 profiles: - profile1 ports: - name: some-port port: 1234 protocol: TCP - name: another-port port: 5432 protocol: UDP ``` ## Definitions[​](#definitions) ### Metadata[​](#metadata) | Field | Description | Accepted Values | Schema | Default | | --------- | ------------------------------------------------------------------ | -------------------------------------------------- | ------ | --------- | | name | The name of this workload endpoint resource. Required. | Alphanumeric string with optional `.`, `_`, or `-` | string | | | namespace | Namespace provides an additional qualification to a resource name. | | string | "default" | | labels | A set of labels to apply to this endpoint. | | map | | ### Spec[​](#spec) | Field | Description | Accepted Values | Schema | Default | | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | ---------------------------------------------- | ------- | | workload | The name of the workload to which this endpoint belongs. | | string | | | orchestrator | The orchestrator that created this endpoint. | | string | | | node | The node where this endpoint resides. | | string | | | containerID | The CNI CONTAINER\_ID of the workload endpoint. | | string | | | pod | Kubernetes pod name for this workload endpoint. | | string | | | endpoint | Container network interface name. | | string | | | ipNetworks | The CIDRs assigned to the interface. | | List of strings | | | ipNATs | List of 1:1 NAT mappings to apply to the endpoint. | | List of [IPNATs](#ipnat) | | | awsElasticIPs | List of AWS Elastic IP addresses that should be considered for this workload; only used for workloads in an AWS-backed IP pool. This should be set via the `cni.projectcalico.org/awsElasticIPs` Pod annotation. | | List of valid IP addresses | | | ipv4Gateway | The gateway IPv4 address for traffic from the workload. | | string | | | ipv6Gateway | The gateway IPv6 address for traffic from the workload. | | string | | | profiles | List of profiles assigned to this endpoint. | | List of strings | | | interfaceName | The name of the host-side interface attached to the workload. | | string | | | mac | The source MAC address of traffic generated by the workload. | | IEEE 802 MAC-48, EUI-48, or EUI-64 | | | ports | List on named ports that this workload exposes. | | List of [WorkloadEndpointPorts](#endpointport) | | ### IPNAT[​](#ipnat) IPNAT contains a single NAT mapping for a WorkloadEndpoint resource. | Field | Description | Accepted Values | Schema | Default | | ---------- | ------------------------------------------- | ------------------ | ------ | ------- | | internalIP | The internal IP address of the NAT mapping. | A valid IP address | string | | | externalIP | The external IP address. | A valid IP address | string | | ### EndpointPort[​](#endpointport) A WorkloadEndpointPort associates a name with a particular TCP/UDP/SCTP port of the endpoint, allowing it to be referenced as a named port in [policy rules](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy#entityrule). | Field | Description | Accepted Values | Schema | Default | | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------- | ------ | ------- | | name | The name to attach to this port, allowing it to be referred to in [policy rules](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy#entityrule). Names must be unique within an endpoint. | | string | | | protocol | The protocol of this named port. | `TCP`, `UDP`, `SCTP` | string | | | port | The workload port number. | `1`-`65535` | int | | | hostPort | Port on the host that is forwarded to this port. | `1`-`65535` | int | | | hostIP | IP address on the host on which the hostPort is accessible. | `1`-`65535` | int | | > **SECONDARY:** On their own, WorkloadEndpointPort entries don't result in any change to the connectivity of the port. They only have an effect if they are referred to in policy. > **SECONDARY:** The hostPort and hostIP fields are read-only and determined from Kubernetes hostPort configuration. These fields are used only when host ports are enabled in Calico. ## Supported operations[​](#supported-operations) | Datastore type | Create/Delete | Update | Get/List | Notes | | --------------------- | ------------- | ------ | -------- | -------------------------------------------------------- | | Kubernetes API server | No | Yes | Yes | WorkloadEndpoints are directly tied to a Kubernetes pod. | ### Architecture ## [📄️ 'The Calico Cloud data path: IP routing and iptables'](https://docs.tigera.io/calico-cloud/reference/architecture/data-path) [Learn how packets flow between workloads in a datacenter, or between a workload and the internet.](https://docs.tigera.io/calico-cloud/reference/architecture/data-path) ## [🗃️ Network design](https://docs.tigera.io/calico-cloud/reference/architecture/design/) [2 items](https://docs.tigera.io/calico-cloud/reference/architecture/design/) ### 'The Calico Cloud data path: IP routing and iptables' One of Calico Cloud’s key features is how packets flow between workloads in a data center, or between a workload and the Internet, without additional encapsulation. In the Calico Cloud approach, IP packets to or from a workload are routed and firewalled by the Linux routing table and iptables or eBPF infrastructure on the workload’s host. For a workload that is sending packets, Calico Cloud ensures that the host is always returned as the next hop MAC address regardless of whatever routing the workload itself might configure. For packets addressed to a workload, the last IP hop is that from the destination workload’s host to the workload itself. ![Calico datapath](https://docs.tigera.io/assets/images/calico-datapath-164f6f29c7b21889c1d4b517a2695533.png) Suppose that IPv4 addresses for the workloads are allocated from a datacenter-private subnet of 10.65/16, and that the hosts have IP addresses from 172.18.203/24. If you look at the routing table on a host: ```bash route -n ``` You will see something like this: ```text Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.18.203.1 0.0.0.0 UG 0 0 0 eth0 10.65.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ns-db03ab89-b4 10.65.0.21 172.18.203.126 255.255.255.255 UGH 0 0 0 eth0 10.65.0.22 172.18.203.129 255.255.255.255 UGH 0 0 0 eth0 10.65.0.23 172.18.203.129 255.255.255.255 UGH 0 0 0 eth0 10.65.0.24 0.0.0.0 255.255.255.255 UH 0 0 0 tapa429fb36-04 172.18.203.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 ``` There is one workload on this host with IP address 10.65.0.24, and accessible from the host via a TAP (or veth, etc.) interface named tapa429fb36-04. Hence there is a direct route for 10.65.0.24, through tapa429fb36-04. Other workloads, with the .21, .22 and .23 addresses, are hosted on two other hosts (172.18.203.126 and .129), so the routes for those workload addresses are via those hosts. The direct routes are set up by a Calico Cloud agent named Felix when it is asked to provision connectivity for a particular workload. A BGP client (such as BIRD) then notices those and distributes them – perhaps via a route reflector – to BGP clients running on other hosts, and hence the indirect routes appear also. ## Is that all?[​](#is-that-all) As far as the static data path is concerned, yes. It’s just a combination of responding to workload ARP requests with the host MAC, IP routing and iptables or eBPF. There’s a great deal more to Calico Cloud in terms of how the required routing and security information is managed, and for handling dynamic things such as workload migration – but the basic data path really is that simple. ### Network design ## [📄️ Calico over Ethernet fabrics](https://docs.tigera.io/calico-cloud/reference/architecture/design/l2-interconnect-fabric) [Understand the interconnect fabric options in a Calico network.](https://docs.tigera.io/calico-cloud/reference/architecture/design/l2-interconnect-fabric) ## [📄️ Calico over IP fabrics](https://docs.tigera.io/calico-cloud/reference/architecture/design/l3-interconnect-fabric) [Understand considerations for implementing interconnect fabrics with Calico.](https://docs.tigera.io/calico-cloud/reference/architecture/design/l3-interconnect-fabric) ### Calico over Ethernet fabrics This is the first of a few *tech notes* that I will be authoring that will discuss some of the various interconnect fabric options in a Calico Cloud network. Any technology that is capable of transporting IP packets can be used as the interconnect fabric in a Calico Cloud network (the first person to test and publish the results of using [IP over Avian Carrier](https://datatracker.ietf.org/doc/html/rfc1149) as a transport for Calico Cloud will earn a very nice dinner on or with the core Calico Cloud team). This means that the standard tools used to transport IP, such as MPLS and Ethernet can be used in a Calico Cloud network. In this note, I'm going to focus on Ethernet as the interconnect network. Talking to most at-scale cloud operators, they have converted to IP fabrics, and as will cover in the next blog post that infrastructure will work for Calico Cloud as well. However, the concerns that drove most of those operators to IP as the interconnection network in their pods are largely ameliorated by Project Calico, allowing Ethernet to be viably considered as a Calico Cloud interconnect, even in large-scale deployments. ## Concerns over Ethernet at scale[​](#concerns-over-ethernet-at-scale) It has been acknowledged by the industry for years that, beyond a certain size, classical Ethernet networks are unsuitable for production deployment. Although there have been [multiple](https://en.wikipedia.org/wiki/Provider_Backbone_Bridge_Traffic_Engineering) [attempts](https://web.archive.org/web/20150923231827/https://www.cisco.com/web/about/ac123/ac147/archived_issues/ipj_14-3/143_trill.html) [to address](https://en.wikipedia.org/wiki/Virtual_Private_LAN_Service) these issues, the scale-out networking community has, largely abandoned Ethernet for anything other than providing physical point-to-point links in the networking fabric. The principal reasons for Ethernet failures at large scale are: 1. Large numbers of *end points* [1](#user-content-fn-1). Each switch in an Ethernet network must learn the path to all Ethernet endpoints that are connected to the Ethernet network. Learning this amount of state can become a substantial task when we are talking about hundreds of thousands of *end points*. 2. High rate of *churn* or change in the network. With that many end points, most of them being ephemeral (such as virtual machines or containers), there is a large amount of *churn* in the network. That load of re-learning paths can be a substantial burden on the control plane processor of most Ethernet switches. 3. High volumes of broadcast traffic. As each node on the Ethernet network must use Broadcast packets to locate peers, and many use broadcast for other purposes, the resultant packet replication to each and every end point can lead to *broadcast storms* in large Ethernet networks, effectively consuming most, if not all resources in the network and the attached end points. 4. Spanning tree. Spanning tree is the protocol used to keep an Ethernet network from forming loops. The protocol was designed in the era of smaller, simpler networks, and it has not aged well. As the number of links and interconnects in an Ethernet network goes up, many implementations of spanning tree become more *fragile*. Unfortunately, when spanning tree fails in an Ethernet network, the effect is a catastrophic loop or partition (or both) in the network, and, in most cases, difficult to troubleshoot or resolve. While many of these issues are crippling at *VM scale* (tens of thousands of end points that live for hours, days, weeks), they will be absolutely lethal at *container scale* (hundreds of thousands of end points that live for seconds, minutes, days). If you weren't ready to turn off your Ethernet data center network before this, I bet you are now. Before you do, however, let's look at how Project Calico can mitigate these issues, even in very large deployments. ## How does Calico Cloud tame the Ethernet daemons?[​](#how-does-calico-cloud-tame-the-ethernet-daemons) First, let's look at how Calico Cloud uses an Ethernet interconnect fabric. It's important to remember that an Ethernet network *sees* nothing on the other side of an attached IP router, the Ethernet network just *sees* the router itself. This is why Ethernet switches can be used at Internet peering points, where large fractions of Internet traffic is exchanged. The switches only see the routers from the various ISPs, not those ISPs' customers' nodes. We leverage the same effect in Calico Cloud. To take the issues outlined above, let's revisit them in a Calico Cloud context. 1. Large numbers of end points. In a Calico Cloud network, the Ethernet interconnect fabric only sees the routers/compute servers, not the end point. In a standard cloud model, where there is tens of VMs per server (or hundreds of containers), this reduces the number of nodes that the Ethernet sees (and has to learn) by one to two orders of magnitude. Even in very large pods (say twenty thousand servers), the Ethernet network would still only see a few tens of thousands of end points. Well within the scale of any competent data center Ethernet top of rack (ToR) switch. 2. High rate of *churn*. In a classical Ethernet data center fabric, there is a *churn* event each time an end point is created, destroyed, or moved. In a large data center, with hundreds of thousands of endpoints, this *churn* could run into tens of events per second, every second of the day, with peaks easily in the hundreds or thousands of events per second. In a Calico Cloud network, however, the *churn* is very low. The only event that would lead to *churn* in a Calico Cloud network's Ethernet fabric would be the addition or loss of a compute server, switch, or physical connection. In a twenty thousand server pod, even with a 5% daily failure rate (a few orders of magnitude more than what is normally experienced), there would only be two thousand events per **day**. Any switch that can not handle that volume of change in the network should not be used for any application. 3. High volume of broadcast traffic. Since the first (and last) hop for any traffic in a Calico Cloud network is an IP hop, and IP hops terminate broadcast traffic, there is no endpoint broadcast network in the Ethernet fabric, period. In fact, the only broadcast traffic that should be seen in the Ethernet fabric is the ARPs of the compute servers locating each other. If the traffic pattern is fairly consistent, the steady-state ARP rate should be almost zero. Even in a pathological case, the ARP rate should be well within normal accepted boundaries. 4. Spanning tree. Depending on the architecture chosen for the Ethernet fabric, it may even be possible to turn off spanning tree. However, even if it is left on, due to the reduction in node count, and reduction in churn, most competent spanning tree implementations should be able to handle the load without stress. With these considerations in mind, it should be evident that an Ethernet connection fabric in Calico Cloud is not only possible, it is practical and should be seriously considered as the interconnect fabric for a Calico Cloud network. As mentioned in the IP fabric post, an IP fabric is also quite feasible for Calico Cloud, but there are more considerations that must be taken into account. The Ethernet fabric option has fewer architectural considerations in its design. ## A brief note about Ethernet topology[​](#a-brief-note-about-ethernet-topology) As mentioned elsewhere in the Calico Cloud documentation, since Calico Cloud can use most of the standard IP tooling, some interesting options regarding fabric topology become possible. We assume that an Ethernet fabric for Calico Cloud would most likely be constructed as a *leaf/spine* architecture. Other options are possible, but the *leaf/spine* is the predominant architectural model in use in scale-out infrastructure today. Since Calico Cloud is an IP routed fabric, a Calico Cloud network can use [ECMP](https://en.wikipedia.org/wiki/Equal-cost_multi-path_routing) to distribute traffic across multiple links (instead of using Ethernet techniques such as MLAG). By leveraging ECMP load balancing on the Calico Cloud compute servers, it is possible to build the fabric out of multiple *independent* leaf/spine planes using no technologies other than IP routing in the Calico Cloud nodes, and basic Ethernet switching in the interconnect fabric. These planes would operate completely independently and could be designed such that they would not share a fault domain. This would allow for the catastrophic failure of one (or more) plane(s) of Ethernet interconnect fabric without the loss of the pod (the failure would just decrease the amount of interconnect bandwidth in the pod). This is a gentler failure mode than the pod-wide IP or Ethernet failure that is possible with today's designs. A more in-depth discussion is possible, so if you'd like, please make a request, and I will put up a post or white paper. In the meantime, it may be interesting to venture over to Facebook's [blog post](https://engineering.fb.com/2014/11/14/production-engineering/introducing-data-center-fabric-the-next-generation-facebook-data-center-network/) on their fabric approach. A quick picture to visualize the idea is shown below. ![A diagram showing the Ethernet spine planes. Each color represents a distinct Ethernet network, transporting a unique IP network.](https://docs.tigera.io/assets/images/l2-spine-planes-d1685acaabb4c4a56f5b79d9932f8796.png) I am not showing the end points in this diagram, and the end points would be unaware of anything in the fabric (as noted above). In the particular case of this diagram, each ToR is segmented into four logical switches (possibly by using 'port VLANs'), [2](#user-content-fn-2) and each compute server has a connection to each of those logical switches. We will identify those logical switches by their color. Each ToR would then have a blue, green, orange, and red logical switch. Those 'colors' would be members of a given *plane*, so there would be a blue plane, a green plane, an orange plane, and a red plane. Each plane would have a dedicated spine switch. and each ToR in a given spine would be connected to its spine, and only its spine. Each plane would constitute an IP network, so the blue plane would be 2001:db8:1000::/36, the green would be 2001:db8:2000::/36, and the orange and red planes would be 2001:db8:3000::/36 and 2001:db8:4000::/36 respectively. [3](#user-content-fn-3) Each IP network (plane) requires its own BGP route reflectors. Those route reflectors need to be peered with each other within the plane, but the route reflectors in each plane do not need to be peered with one another. Therefore, a fabric of four planes would have four route reflector meshes. Each compute server, border router, *etc.* would need to be a route reflector client of at least one route reflector in each plane, and very preferably two or more in each plane. A diagram that visualizes the route reflector environment can be found below. ![A diagram showing the route reflector topology in the l2 spine plane architecture. The dashed diamonds are the route reflectors, with one or more per L2 spine plane. All compute servers are peered to all route reflectors, and all the route reflectors in a given plane are also meshed. However, the route reflectors in each spine plane are not meshed together (e.g. the blue route reflectors are not peered or meshed with the red route reflectors. The route reflectors themselves could be daemons running on the actual compute servers or on other dedicated or networking hardware.](https://docs.tigera.io/assets/images/l2-rr-spine-planes-d10ad67fe16f2c08329e0baf80f213fc.png) These route reflectors could be dedicated hardware connected to the spine switches (or the spine switches themselves), or physical or virtual route reflectors connected to the necessary logical leaf switches (blue, green, orange, and red). That may be a route reflector running on a compute server and connected directly to the correct plane link, and not routed through the vRouter, to avoid the chicken and egg problem that would occur if the route reflector were "behind" the Calico Cloud network. Other physical and logical configurations and counts are, of course, possible, this is just an example. The logical configuration would then have each compute server would have an address on each plane's subnet, and announce its end points on each subnet. If ECMP is then turned on, the compute servers would distribute the load across all planes. If a plane were to fail (say due to a spanning tree failure), then only that one plane would fail. The remaining planes would stay running. ## Footnotes[​](#footnote-label) 1. In this document (and in all Calico Cloud documents) we tend to use the terms *end point* to refer to a virtual machine, container, appliance, bare metal server, or any other entity that is connected to a Calico Cloud network. If we are referring to a specific type of end point, we will call that out (such as referring to the behavior of VMs as distinct from containers). [↩](#user-content-fnref-1) 2. We are using logical switches in this example. Physical ToRs could also be used, or a mix of the two (say 2 logical switches hosted on each physical switch). [↩](#user-content-fnref-2) 3. We use IPv6 here purely as an example. IPv4 would be configured similarly. I welcome your questions, either here on the blog, or via the Project Calico mailing list. [↩](#user-content-fnref-3) ### Calico over IP fabrics Calico Cloud provides an end-to-end IP network that interconnects the endpoints [1](#user-content-fn-1) in a scale-out or cloud environment. To do that, it needs an *interconnect fabric* to provide the physical networking layer on which Calico Cloud operates [2](#user-content-fn-2). While Calico Cloud is designed to work with any underlying interconnect fabric that can support IP traffic, the fabric that has the least considerations attached to its implementation is an Ethernet fabric as discussed in our earlier [technical note](https://docs.tigera.io/calico-cloud/reference/architecture/design/l2-interconnect-fabric). In most cases, the Ethernet fabric is the appropriate choice, but there are infrastructures where L3 (an IP fabric) has already been deployed, or will be deployed, and it makes sense for Calico Cloud to operate in those environments. However, since Calico Cloud is, itself, a routed infrastructure, there are more engineering, architecture, and operations considerations that have to be weighed when running Calico Cloud with an IP routed interconnection fabric. We will briefly outline those in the rest of this post. That said, Calico Cloud operates equally well with Ethernet or IP interconnect fabrics. ## Background[​](#background) ### Basic Calico Cloud architecture overview[​](#basic-calico-cloud-architecture-overview) In a Calico Cloud network, each compute server acts as a router for all of the endpoints that are hosted on that compute server. We call that function a vRouter. The data path is provided by the Linux kernel, the control plane by a BGP protocol server, and management plane by Calico Cloud's on-server agent, *Felix*. Each endpoint can only communicate through its local vRouter, and the first and last *hop* in any Calico Cloud packet flow is an IP router hop through a vRouter. Each vRouter announces all of the endpoints it is attached to all the other vRouters and other routers on the infrastructure fabric, using BGP, usually with BGP route reflectors to increase scale. A discussion of why we use BGP can be found in the [Why BGP?](https://www.projectcalico.org/why-bgp/) blog post. Access control lists (ACLs) enforce security (and other) policy as directed by whatever cloud orchestrator is in use. There are other components in the Calico Cloud architecture, but they are irrelevant to the interconnect network fabric discussion. ### Overview of current common IP scale-out fabric architectures[​](#overview-of-current-common-ip-scale-out-fabric-architectures) There are two approaches to building an IP fabric for a scale-out infrastructure. However, all of them, to date, have assumed that the edge router in the infrastructure is the top of rack (TOR) switch. In the Calico Cloud model, that function is pushed to the compute server itself. The two approaches are outlined below, in this technical note, we will cover the second option, as it is more common in the scale-out world. If there is interest in the first approach, please contact Project Calico, and we can discuss, and if there is enough interest, maybe we will do another technical note on that approach. If you know of other approaches in use, we would be happy to host a guest technical note. 1. The routing infrastructure is based on some form of IGP. Due to the limitations in scale of IGP networks (see the [why BGP post](https://www.projectcalico.org/why-bgp/) for discussion of this topic), the Project Calico team does not believe that using an IGP to distribute endpoint reachability information will adequately scale in a Calico Cloud environment. However, it is possible to use a combination of IGP and BGP in the interconnect fabric, where an IGP communicates the path to the *next-hop* router (in Calico Cloud, this is often the destination compute server) and BGP is used to distribute the actual next-hop for a given endpoint. This is a valid model, and, in fact is the most common approach in a widely distributed IP network (say a carrier's backbone network). The design of these networks is somewhat complex though, and will not be addressed further in this technical note. [3](#user-content-fn-3) 2. The other model, and the one that this note concerns itself with, is one where the routing infrastructure is based entirely on BGP. In this model, the IP network is "tight enough" or has a small enough diameter that BGP can be used to distribute endpoint routes, and the paths to the next-hops for those routes is known to all of the routers in the network (in a Calico Cloud network this includes the compute servers). This is the network model that this note will address. ### BGP-only interconnect fabrics[​](#bgp-only-interconnect-fabrics) There are multiple methods to build a BGP-only interconnect fabric. We will focus on three models, each with two widely viable variations. There are other options, and we will briefly touch on why we didn't include some of them in the [Other Options appendix](#other-options). The two methods are: 1. A BGP fabric where each of the TOR switches (and their subsidiary compute servers) are a unique [Autonomous System (AS)](https://en.wikipedia.org/wiki/Autonomous_System_\(Internet\)) and they are interconnected via either an Ethernet switching plane provided by the spine switches in a [leaf/spine](http://bradhedlund.com/2012/10/24/video-a-basic-introduction-to-the-leafspine-data-center-networking-fabric-design/) architecture, or via a set of spine switches, each of which is also a unique AS. We'll refer to this as the *AS per rack* model. This model is detailed in [IETF RFC 7938](https://datatracker.ietf.org/doc/html/rfc7938). 2. A BGP fabric where each of the compute servers is a unique AS, and the TOR switches make up a transit AS. We'll refer to this as the *AS per server* model. Each of these models can either have an Ethernet or IP spine. In the case of an Ethernet spine, each spine switch provides an isolated Ethernet connection *plane* as in the Calico Cloud Ethernet interconnect fabric model and each TOR switch is connected to each spine switch. Another model is where each spine switch is a unique AS, and each TOR switch BGP peers with each spine switch. In both cases, the TOR switches use ECMP to load-balance traffic between all available spine switches. ### Some BGP network design considerations[​](#some-bgp-network-design-considerations) Contrary to popular opinion, BGP is actually a fairly simple protocol. For example, the BGP configuration on a Calico Cloud compute server is approximately sixty lines long, not counting comments. The perceived complexity is due to the things that you can *do* with BGP. Many uses of BGP involve complex policy rules, where the behavior of BGP can be modified to meet technical (or business, financial, political, *etc.*) requirements. A default Calico Cloud network does not venture into those areas, [4](#user-content-fn-4) and therefore is fairly straight forward. That said, there are a few design rules for BGP that need to be kept in mind when designing an IP fabric that will interconnect nodes in a Calico Cloud network. These BGP design requirements *can* be worked around, if necessary, but doing so takes the designer out of the standard BGP *envelope* and should only be done by an implementer who is *very* comfortable with advanced BGP design. These considerations are: AS continuity : or *AS puddling* Any router in an AS *must* be able to communicate with any other router in that same AS without transiting another AS. Next hop behavior : By default BGP routers do not change the *next hop* of a route if it is peering with another router in its same AS. The inverse is also true, a BGP router will set itself as the *next hop* of a route if it is peering with a router in another AS. Route reflection : All BGP routers in a given AS must *peer* with all the other routers in that AS. This is referred to a *complete BGP mesh*. This can become problematic as the number of routers in the AS scales up. The use of *route reflectors* reduce the need for the complete BGP mesh. However, route reflectors also have scaling considerations. Endpoints : In a Calico Cloud network, each endpoint is a route. Hardware networking platforms are constrained by the number of routes they can learn. This is usually in range of 10,000's or 100,000's of routes. Route aggregation can help, but that is usually dependent on the capabilities of the scheduler used by the orchestration software. A deeper discussion of these considerations can be found in the IP Fabric Design Considerations\_ appendix. The designs discussed below address these considerations. ### The *AS Per Rack* model[​](#the-as-per-rack-model) This model is the closest to the model suggested by [IETF RFC 7938](https://datatracker.ietf.org/doc/html/rfc7938). As mentioned earlier, there are two versions of this model, one with an set of Ethernet planes interconnecting the ToR switches, and the other where the core planes are also routers. The following diagrams may be useful for the discussion. ![Diagram showing the AS per rack model with ToR switches meshed via Ethernet switching planes at the spine layer](https://docs.tigera.io/assets/images/l3-fabric-diagrams-as-rack-l2-spine-586a942656c4718cae0d17e78de81a15.png) The diagram above shows the *AS per rack model* where the ToR switches are physically meshed via a set of Ethernet switching planes. ![Diagram showing the AS per rack model with ToR switches meshed via discrete BGP spine routers, each in their own AS](https://docs.tigera.io/assets/images/l3-fabric-diagrams-as-rack-l3-spine-731d38ec8419d6e7a50a6ee9a610bdf1.png) The diagram above shows the *AS per rack model* where the ToR switches are physically meshed via a set of discrete BGP spine routers, each in their own AS. In this approach, every ToR-ToR or ToR-Spine (in the case of an AS per spine) link is an eBGP peering which means that there is no route-reflection possible (using standard BGP route reflectors) *north* of the ToR switches. If the L2 spine option is used, the result of this is that each ToR must either peer with every other ToR switch in the cluster (which could be hundreds of peers). If the AS per spine option is used, then each ToR only has to peer with each spine (there are usually somewhere between two and sixteen spine switches in a pod). However, the spine switches must peer with all ToR switches (again, that would be hundreds, but most spine switches have more control plane capacity than the average ToR, so this might be more scalable in many circumstances). Within the rack, the configuration is the same for both variants, and is somewhat different than the configuration north of the ToR. Every router within the rack, which, in the case of Calico Cloud is every compute server, shares the same AS as the ToR that they are connected to. That connection is in the form of an Ethernet switching layer. Each router in the rack must be directly connected to enable the AS to remain contiguous. The ToR's *router* function is then connected to that Ethernet switching layer as well. The actual configuration of this is dependent on the ToR in use, but usually it means that the ports that are connected to the compute servers are treated as *subnet* or *segment* ports, and then the ToR's *router* function has a single interface into that subnet. This configuration allows each compute server to connect to each other compute server in the rack without going through the ToR router, but it will, of course, go through the ToR switching function. The compute servers and the ToR router could all be directly meshed, or a route reflector could be used within the rack, either hosted on the ToR itself, or as a virtual function hosted on one or more compute servers within the rack. The ToR, as the eBGP router redistributes all of the routes from other ToRs as well as routes external to the data center to the compute servers that are in its AS, and announces all of the routes from within the AS (rack) to the other ToRs and the larger world. This means that each compute server will see the ToR as the next hop for all external routes, and the individual compute servers are the next hop for all routes internal to the rack. ### The *AS per Compute Server* model[​](#the-as-per-compute-server-model) This model takes the concept of an AS per rack to its logical conclusion. In the earlier referenced [IETF RFC 7938](https://datatracker.ietf.org/doc/html/rfc7938) the assumption in the overall model is that the ToR is first tier aggregating and routing element. In Calico Cloud, the ToR, if it is an L3 router, is actually the second tier. Remember, in Calico Cloud, the compute server is always the first/last router for an endpoint, and is also the first/last point of aggregation. Therefore, if we follow the architecture of the draft, the compute server, not the ToR should be the AS boundary. The differences can be seen in the following two diagrams. ![Diagram showing the AS per compute server model with ToR switches meshed via Ethernet switching planes at the spine layer](https://docs.tigera.io/assets/images/l3-fabric-diagrams-as-server-l2-spine-ef320fdea22b2f69da6211d3731a3c32.png) The diagram above shows the *AS per compute server model* where the ToR switches are physically meshed via a set of Ethernet switching planes. ![Diagram showing the AS per compute server model with ToR switches connected to independent routing planes at the spine layer](https://docs.tigera.io/assets/images/l3-fabric-diagrams-as-server-l3-spine-0515c7852f8f7aaf4d550012ff10b5fe.png) The diagram above shows the *AS per compute server model* where the ToR switches are physically connected to a set of independent routing planes. As can be seen in these diagrams, there are still the same two variants as in the *AS per rack* model, one where the spine switches provide a set of independent Ethernet planes to interconnect the ToR switches, and the other where that is done by a set of independent routers. The real difference in this model, is that the compute servers as well as the ToR switches are all independent autonomous systems. To make this work at scale, the use of four byte AS numbers as discussed in [RFC 4893](http://www.faqs.org/rfcs/rfc4893.html). Without using four byte AS numbering, the total number of ToRs and compute servers in a Calico Cloud fabric would be limited to the approximately five thousand available private AS [5](#user-content-fn-5) numbers. If four byte AS numbers are used, there are approximately ninety-two million private AS numbers available. This should be sufficient for any given Calico Cloud fabric. The other difference in this model *vs.* the AS per rack model, is that there are no route reflectors used, as all BGP peerings are eBGP. In this case, each compute server in a given rack peers with its ToR switch which is also acting as an eBGP router. For two servers within the same rack to communicate, they will be routed through the ToR. Therefore, each server will have one peering to each ToR it is connected to, and each ToR will have a peering with each compute server that it is connected to (normally, all the compute servers in the rack). The inter-ToR connectivity considerations are the same in scale and scope as in the AS per rack model. ### The *Downward Default* model[​](#the-downward-default-model) The final model is a bit different. Whereas, in the previous models, all of the routers in the infrastructure carry full routing tables, and leave their AS paths intact, this model [6](#user-content-fn-6) removes the AS numbers at each stage of the routing path. This is to prevent routes from other nodes in the network from not being installed due to it coming from the *local* AS (since they share the source and dest of the route share the same AS). The following diagram will show the AS relationships in this model. ![Diagram showing the downward default model where all Calico nodes share one AS and all ToR switches share another, with spine routers announcing default routes downward](https://docs.tigera.io/assets/images/l3-fabric-downward-default-30bce2fe705b14f16d7381cf5612a81c.png) In the diagram above, we are showing that all Calico Cloud nodes share the same AS number, as do all ToR switches. However, those ASs are different (*A1* is not the same network as *A2*, even though the both share the same AS number *A* ). While the use of a single AS for all ToR switches, and another for all compute servers simplifies deployment (standardized configuration), the real benefit comes in the offloading of the routing tables in the ToR switches. In this model, each router announces all of its routes to its upstream peer (the Calico Cloud routers to their ToR, the ToRs to the spine switches). However, in return, the upstream router only announces a default route. In this case, a given Calico Cloud router only has routes for the endpoints that are locally hosted on it, as well as the default from the ToR. Since the ToR is the only route for the Calico Cloud network the rest of the network, this matches reality. The same happens between the ToR switches and the spine. This means that the ToR only has to install the routes that are for endpoints that are hosted on its downstream Calico Cloud nodes. Even if we were to host 200 endpoints per Calico Cloud node, and stuff 80 Calico Cloud nodes in each rack, that would still limit the routing table on the ToR to a maximum of 16,000 entries (well within the capabilities of even the most modest of switches). Since the default is originated by the Spine (originally) there is no chance for a downward announced route to originate from the recipient's AS, preventing the *AS puddling* problem. There is one (minor) drawback to this model, in that all traffic that is destined for an invalid destination (the destination IP does not exist) will be forwarded to the spine switches before they are dropped. It should also be noted that the spine switches do need to carry all of the Calico Cloud network routes, just as they do in the routed spines in the previous examples. In short, this model imposes no more load on the spines than they already would have, and substantially reduces the amount of routing table space used on the ToR switches. It also reduces the number of routes in the Calico Cloud nodes, but, as we have discussed before, that is not a concern in most deployments as the amount of memory consumed by a full routing table in Calico Cloud is a fraction of the total memory available on a modern compute server. ## Recommendation[​](#recommendation) The Project Calico team recommends the use of the [AS per rack](#the-as-per-rack-model) model if the resultant routing table size can be accommodated by the ToR and spine switches, remembering to account for projected growth. If there is concern about the route table size in the ToR switches, the team recommends the [Downward Default](#the-downward-default-model) model. If there are concerns about both the spine and ToR switch route table capacity, or there is a desire to run a very simple L2 fabric to connect the Calico Cloud nodes, then the user should consider the Ethernet fabric as detailed in [this post](https://docs.tigera.io/calico-cloud/reference/architecture/design/l2-interconnect-fabric). If a Calico Cloud user is interested in the AS per compute server, the Project Calico team would be very interested in discussing the deployment of that model. ## Appendix[​](#appendix) ### Other Options[​](#other-options) The way the physical and logical connectivity is laid out in this note, and the [Ethernet fabric note](https://docs.tigera.io/calico-cloud/reference/architecture/design/l2-interconnect-fabric), The next hop router for a given route is always directly connected to the router receiving that route. This makes the need for another protocol to distribute the next hop routes unnecessary. However, in many (or most) WAN BGP networks, the routers within a given AS may not be directly adjacent. Therefore, a router may receive a route with a next hop address that it is not directly adjacent to. In those cases, an IGP, such as OSPF or IS-IS, is used by the routers within a given AS to determine the path to the BGP next hop route. There may be Calico Cloud architectures where there are similar models where the routers within a given AS are not directly adjacent. In those models, the use of an IGP in Calico Cloud may be warranted. The configuration of those protocols are, however, beyond the scope of this technical note. ### IP Fabric Design Considerations[​](#ip-fabric-design-considerations) #### AS puddling[​](#as-puddling) The first consideration is that an AS must be kept contiguous. This means that any two nodes in a given AS must be able to communicate without traversing any other AS. If this rule is not observed, the effect is often referred to as *AS puddling* and the network will *not* function correctly. A corollary of that rule is that any two administrative regions that share the same AS number, are in the same AS, even if that was not the desire of the designer. BGP has no way of identifying if an AS is local or foreign other than the AS number. Therefore re-use of an AS number for two *networks* that are not directly connected, but only connected through another *network* or AS number will not work without a lot of policy changes to the BGP routers. Another corollary of that rule is that a BGP router will not propagate a route to a peer if the route has an AS in its path that is the same AS as the peer. This prevents loops from forming in the network. The effect of this prevents two routers in the same AS from transiting another router (either in that AS or not). #### Next hop behavior[​](#next-hop-behavior) Another consideration is based on the differences between iBGP and eBGP. BGP operates in two modes, if two routers are BGP peers, but share the same AS number, then they are considered to be in an *internal* BGP (or iBGP) peering relationship. If they are members of different AS's, then they are in an *external* or eBGP relationship. BGP's original design model was that all BGP routers within a given AS would know how to get to one another (via static routes, IGP [7](#user-content-fn-7) routing protocols, or the like), and that routers in different ASs would not know how to reach one another unless they were directly connected. Based on that design point, routers in an iBGP peering relationship assume that they do not transit traffic for other iBGP routers in a given AS (i.e. A can communicate with C, and therefore will not need to route through B), and therefore, do not change the *next hop* attribute in BGP [8](#user-content-fn-8). A router with an eBGP peering, on the other hand, assumes that its eBGP peer will not know how to reach the next hop route, and then will substitute its own address in the next hop field. This is often referred to as *next hop self*. In the Calico Cloud [Ethernet fabric](https://docs.tigera.io/calico-cloud/reference/architecture/design/l2-interconnect-fabric) model, all of the compute servers (the routers in a Calico Cloud network) are directly connected over one or more Ethernet network(s) and therefore are directly reachable. In this case, a router in the Calico Cloud network does not need to set *next hop self* within the Calico Cloud fabric. The models we present in this technical note insure that all routes that may traverse a non-Calico Cloud router are eBGP routes, and therefore *next hop self* is automatically set correctly. If a deployment of Calico Cloud in an IP interconnect fabric does not satisfy that constraint, then *next hop self* must be appropriately configured. #### Route reflection[​](#route-reflection) As mentioned above, BGP expects that all of the iBGP routers in a network can see (and speak) directly to one another, this is referred to as a *BGP full mesh*. In small networks this is not a problem, but it does become interesting as the number of routers increases. For example, if you have 99 BGP routers in an AS and wish to add one more, you would have to configure the peering to that new router on each of the 99 existing routers. Not only is this a problem at configuration time, it means that each router is maintaining 100 protocol adjacencies, which can start being a drain on constrained resources in a router. While this might be *interesting* at 100 routers, it becomes an impossible task with 1000's or 10,000's of routers (the potential size of a Calico Cloud network). Conveniently, large scale/Internet scale networks solved this problem almost 20 years ago by deploying BGP route reflection as described in [RFC 1966](http://www.faqs.org/rfcs/rfc1966.html). This is a technique supported by almost all BGP routers today. In a large network, a number of route reflectors [9](#user-content-fn-9) are evenly distributed and each iBGP router is *peered* with one or more route reflectors (usually 2 or 3). Each route reflector can handle 10's or 100's of route reflector clients (in Calico Cloud's case, the compute server), depending on the route reflector being used. Those route reflectors are, in turn, peered with each other. This means that there are an order of magnitude less route reflectors that need to be completely meshed, and each route reflector client is only configured to peer to 2 or 3 route reflectors. This is much easier to manage. Other route reflector architectures are possible, but those are beyond the scope of this document. #### Endpoints[​](#endpoints) The final consideration is the number of endpoints in a Calico Cloud network. In the [Ethernet fabric](https://docs.tigera.io/calico-cloud/reference/architecture/design/l2-interconnect-fabric) case the number of endpoints is not constrained by the interconnect fabric, as the interconnect fabric does not *see* the actual endpoints, it only *sees* the actual vRouters, or compute servers. This is not the case in an IP fabric, however. IP networks forward by using the destination IP address in the packet, which, in Calico Cloud's case, is the destination endpoint. That means that the IP fabric nodes (ToR switches and/or spine switches, for example) must know the routes to each endpoint in the network. They learn this by participating as route reflector clients in the BGP mesh, just as the Calico Cloud vRouter/compute server does. However, unlike a compute server which has a relatively unconstrained amount of memory, a physical switch is either memory constrained, or quite expensive. This means that the physical switch has a limit on how many *routes* it can handle. The current industry standard for modern commodity switches is in the range of 128,000 routes. This means that, without other routing *tricks*, such as aggregation, a Calico Cloud installation that uses an IP fabric will be limited to the routing table size of its constituent network hardware, with a reasonable upper limit today of 128,000 endpoints. ## Footnotes[​](#footnote-label) 1. In Calico Cloud's terminology, an endpoint is an IP address and interface. It could refer to a VM, a container, or even a process bound to an IP address running on a bare metal server. [↩](#user-content-fnref-1) 2. This interconnect fabric provides the connectivity between the Calico Cloud (v)Router (in almost all cases, the compute servers) nodes, as well as any other elements in the fabric (*e.g.* bare metal servers, border routers, and appliances). [↩](#user-content-fnref-2) 3. If there is interest in a discussion of this approach, please let us know. The Project Calico team could either arrange a discussion, or if there was enough interest, publish a follow-up tech note. [↩](#user-content-fnref-3) 4. However those tools are available if a given Calico Cloud instance needs to utilize those policy constructs. [↩](#user-content-fnref-4) 5. The two byte AS space reserves approximately the last five thousand AS numbers for private use. There is no technical reason why other AS numbers could not be used. However the re-use of global scope AS numbers within a private infrastructure is strongly discouraged. The chance for routing system failure or incorrect routing is substantial, and not restricted to the entity that is doing the reuse. [↩](#user-content-fnref-5) 6. We first saw this design in a customer's lab, and thought it innovative enough to share (we asked them first, of course). Similar *AS Path Stripping* approaches are used in ISP networks, however. [↩](#user-content-fnref-6) 7. An Interior Gateway Protocol is a local routing protocol that does not cross an AS boundary. The primary IGPs in use today are OSPF and IS-IS. While complex iBGP networks still use IGP routing protocols, a data center is normally a fairly simple network, even if it has many routers in it. Therefore, in the data center case, the use of an IGP can often be disposed of. [↩](#user-content-fnref-7) 8. A Next hop is an attribute of a route announced by a routing protocol. In simple terms a route is defined by a *target*, or the destination that is to be reached, and a *next hop*, which is the next router in the path to reach that target. There are many other characteristics in a route, but those are well beyond the scope of this post. [↩](#user-content-fnref-8) 9. A route reflector may be a physical router, a software appliance, or simply a BGP daemon. It only processes routing messages, and does not pass actual data plane traffic. However, some route reflectors are co-resident on regular routers that do pass data plane traffic. While they may sit on one platform, the functions are distinct. [↩](#user-content-fnref-9) ### Component resources ## [📄️ Configuring the Calico Cloud CNI plugins](https://docs.tigera.io/calico-cloud/reference/component-resources/configuration) [Details for configuring the Calico Cloud CNI plugins.](https://docs.tigera.io/calico-cloud/reference/component-resources/configuration) ## [📄️ Configure resource requests and limits](https://docs.tigera.io/calico-cloud/reference/component-resources/configure-resources) [Configure Resource requests and limits.](https://docs.tigera.io/calico-cloud/reference/component-resources/configure-resources) ## [🗃️ Calico Cloud Kubernetes controllers](https://docs.tigera.io/calico-cloud/reference/component-resources/kube-controllers/) [2 items](https://docs.tigera.io/calico-cloud/reference/component-resources/kube-controllers/) ## [🗃️ Calico Cloud node (cnx-node)](https://docs.tigera.io/calico-cloud/reference/component-resources/node/) [2 items](https://docs.tigera.io/calico-cloud/reference/component-resources/node/) ### Configuring the Calico Cloud CNI plugins **Tab: Operator** The Calico Cloud CNI plugins do not need to be configured directly when installed by the operator. For a complete operator configuration reference, see [the installation API reference documentation](https://docs.tigera.io/calico-cloud/reference/installation/api). **Tab: Manifest** The Calico Cloud CNI plugin is configured through the standard CNI [configuration mechanism](https://github.com/containernetworking/cni/blob/master/SPEC.md#network-configuration) A minimal configuration file that uses Calico Cloud for networking and IPAM looks like this ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "ipam": { "type": "calico-ipam" } } ``` If the `cnx-node` container on a node registered with a `NODENAME` other than the node hostname, the CNI plugin on this node must be configured with the same `nodename`: ```json { "name": "any_name", "nodename": "", "type": "calico", "ipam": { "type": "calico-ipam" } } ``` Additional configuration can be added as detailed below. ## Generic[​](#generic) ### Datastore type[​](#datastore-type) The Calico Cloud CNI plugin supports the following datastore: - `datastore_type` (kubernetes) ### Logging[​](#logging) Logging is always to `stderr`. Logs are also written to `/var/log/calico/cni/cni.log` on each host by default. Logging can be configured using the following options in the netconf. | Option name | Default | Description | | -------------------- | ----------------------------- | --------------------------------------------------------------------------------------------------------- | | `log_level` | INFO | Logging level. Allowed levels are `ERROR`, `WARNING`, `INFO`, and `DEBUG`. | | `log_file_path` | `/var/log/calico/cni/cni.log` | Location on each host to write CNI log files to. Logging to file can be disabled by removing this option. | | `log_file_max_size` | 100 | Max file size in MB log files can reach before they are rotated. | | `log_file_max_age` | 30 | Max age in days that old log files will be kept on the host before they are removed. | | `log_file_max_count` | 10 | Max number of rotated log files allowed on the host before they are cleaned up. | ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "log_level": "DEBUG", "log_file_path": "/var/log/calico/cni/cni.log", "ipam": { "type": "calico-ipam" } } ``` ### IPAM[​](#ipam) When using Calico Cloud IPAM, the following flags determine what IP addresses should be assigned. NOTE: These flags are strings and not boolean values. - `assign_ipv4` (default: `"true"`) - `assign_ipv6` (default: `"false"`) A specific IP address can be chosen by using [`CNI_ARGS`](https://github.com/appc/cni/blob/master/SPEC.md#parameters) and setting `IP` to the desired value. By default, Calico Cloud IPAM will assign IP addresses from all the available IP pools. Optionally, the list of possible IPv4 and IPv6 pools can also be specified via the following properties: - `ipv4_pools`: An array of CIDR strings or pool names. (e.g., `"ipv4_pools": ["10.0.0.0/24", "20.0.0.0/16", "default-ipv4-ippool"]`) - `ipv6_pools`: An array of CIDR strings or pool names. (e.g., `"ipv6_pools": ["2001:db8::1/120", "namedpool"]`) Example CNI config: ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "ipam": { "type": "calico-ipam", "assign_ipv4": "true", "assign_ipv6": "true", "ipv4_pools": ["10.0.0.0/24", "20.0.0.0/16", "default-ipv4-ippool"], "ipv6_pools": ["2001:db8::1/120", "default-ipv6-ippool"] } } ``` > **SECONDARY:** `ipv6_pools` will be respected only when `assign_ipv6` is set to `"true"`. Any IP pools specified in the CNI config must have already been created. It is an error to specify IP pools in the config that do not exist. ### Container settings[​](#container-settings) The following options allow configuration of settings within the container namespace. - allow\_ip\_forwarding (default is `false`) ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "ipam": { "type": "calico-ipam" }, "container_settings": { "allow_ip_forwarding": true } } ``` ### Readiness Gates[​](#readiness-gates) The following option makes CNI plugin wait for specified endpoint(s) to be ready before configuring pod networking. - `readiness_gates` This is an optional property that takes an array of URLs. Each URL specified will be polled for readiness and pod networking will continue startup once all readiness\_gates are ready. Example CNI config: ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "ipam": { "type": "calico-ipam" }, "readiness_gates": ["http://localhost:9099/readiness", "http://localhost:8888/status"] } ``` ## Kubernetes specific[​](#kubernetes-specific) When using the Calico Cloud CNI plugin with Kubernetes, the plugin must be able to access the Kubernetes API server to find the labels assigned to the Kubernetes pods. The recommended way to configure access is through a `kubeconfig` file specified in the `kubernetes` section of the network config. e.g. ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "kubernetes": { "kubeconfig": "/path/to/kubeconfig" }, "ipam": { "type": "calico-ipam" } } ``` As a convenience, the API location can also be configured directly, e.g. ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "kubernetes": { "k8s_api_root": "http://127.0.0.1:8080" }, "ipam": { "type": "calico-ipam" } } ``` ### Enabling Kubernetes policy[​](#enabling-kubernetes-policy) If you wish to use the Kubernetes `NetworkPolicy` resource then you must set a policy type in the network config. There is a single supported policy type, `k8s`. When set, you must also run `tigera/kube-controllers` with the policy, profile, and workloadendpoint controllers enabled. ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "policy": { "type": "k8s" }, "kubernetes": { "kubeconfig": "/path/to/kubeconfig" }, "ipam": { "type": "calico-ipam" } } ``` When using `type: k8s`, the Calico Cloud CNI plugin requires read-only Kubernetes API access to the `Pods` resource in all namespaces. ## IPAM[​](#ipam-1) ### Using host-local IPAM[​](#using-host-local-ipam) Calico can be configured to use [host-local IPAM](https://www.cni.dev/plugins/current/ipam/host-local/) instead of the default `calico-ipam`. Host local IPAM uses a pre-determined CIDR per-host, and stores allocations locally on each node. This is in contrast to Calico IPAM, which dynamically allocates blocks of addresses and single addresses alike in response to cluster needs. Host local IPAM is generally only used on clusters where integration with the Kubernetes [route controller](https://kubernetes.io/docs/concepts/architecture/cloud-controller/#route-controller) is necessary. Note that some Calico features - such as the ability to request a specific address or pool for a pod - require Calico IPAM to function, and will not work with host-local IPAM enabled. - Operator - Manifest The `host-local` IPAM plugin can be configured by setting the `Spec.CNI.IPAM.Plugin` field to `HostLocal` on the [operator.tigera.io/Installation](https://docs.tigera.io/calico-cloud/reference/installation/api#installation) API. Calico will use the `host-local` IPAM plugin to allocate IPv4 addresses from the node's IPv4 pod CIDR if there is an IPv4 pool configured in `Spec.IPPools`, and an IPv6 address from the node's IPv6 pod CIDR if there is an IPv6 pool configured in `Spec.IPPools`. The following example configures Calico to assign dual-stack IPs to pods using the host-local IPAM plugin. ```yaml kind: Installation apiVersion: operator.tigera.io/v1 metadata: name: default spec: calicoNetwork: ipPools: - cidr: 192.168.0.0/16 - cidr: 2001:db8::/64 cni: type: Calico ipam: type: HostLocal ``` When using the CNI `host-local` IPAM plugin, two special values - `usePodCidr` and `usePodCidrIPv6` - are allowed for the subnet field (either at the top-level, or in a "range"). This tells the plugin to determine the subnet to use from the Kubernetes API based on the Node.podCIDR field. Calico Cloud does not use the `gateway` field of a range so that field is not required and it will be ignored if present. > **SECONDARY:** `usePodCidr` and `usePodCidrIPv6` can only be used as the value of the `subnet` field, it cannot be used in `rangeStart` or `rangeEnd` so those values are not useful if `subnet` is set to `usePodCidr`. Calico Cloud supports the host-local IPAM plugin's `routes` field as follows: - If there is no `routes` field, Calico Cloud will install a default `0.0.0.0/0`, and/or `::/0` route into the pod (depending on whether the pod has an IPv4 and/or IPv6 address). - If there is a `routes` field then Calico Cloud will program *only* the routes in the routes field into the pod. Since Calico Cloud implements a point-to-point link into the pod, the `gw` field is not required and it will be ignored if present. All routes that Calico Cloud installs will have Calico Cloud's link-local IP as the next hop. Calico Cloud CNI plugin configuration: - `node_name` - The node name to use when looking up the CIDR value (defaults to current hostname) ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "kubernetes": { "kubeconfig": "/path/to/kubeconfig", "node_name": "node-name-in-k8s" }, "ipam": { "type": "host-local", "ranges": [[{ "subnet": "usePodCidr" }], [{ "subnet": "usePodCidrIPv6" }]], "routes": [{ "dst": "0.0.0.0/0" }, { "dst": "2001:db8::/96" }] } } ``` When making use of the `usePodCidr` or `usePodCidrIPv6` options, the Calico Cloud CNI plugin requires read-only Kubernetes API access to the `Nodes` resource. #### Configuring node and typha[​](#configuring-node-and-typha) When using `host-local` IPAM with the Kubernetes API datastore, you must configure both cnx-node and the Typha deployment to use the `Node.podCIDR` field by setting the environment variable `USE_POD_CIDR=true` in each. ### Using Kubernetes annotations[​](#using-kubernetes-annotations) #### Specifying IP pools on a per-namespace or per-pod basis[​](#specifying-ip-pools-on-a-per-namespace-or-per-pod-basis) In addition to specifying IP pools in the CNI config as discussed above, Calico Cloud IPAM supports specifying IP pools per-namespace or per-pod using the following [Kubernetes annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/). - `cni.projectcalico.org/ipv4pools`: A list of configured IPv4 Pools from which to choose an address for the pod. Example: ```yaml annotations: 'cni.projectcalico.org/ipv4pools': '["default-ipv4-ippool"]' ``` - `cni.projectcalico.org/ipv6pools`: A list of configured IPv6 Pools from which to choose an address for the pod. Example: ```yaml annotations: 'cni.projectcalico.org/ipv6pools': '["2001:db8::1/120"]' ``` If provided, these IP pools will override any IP pools specified in the CNI config. > **SECONDARY:** This requires the IP pools to exist before `ipv4pools` or `ipv6pools` annotations are used. Requesting a subset of an IP pool is not supported. IP pools requested in the annotations must exactly match a configured [IPPool](https://docs.tigera.io/calico-cloud/reference/resources/ippool) resource. > **SECONDARY:** The Calico Cloud CNI plugin supports specifying an annotation per namespace. If both the namespace and the pod have this annotation, the pod information will be used. Otherwise, if only the namespace has the annotation the annotation of the namespace will be used for each pod in it. #### Requesting a specific IP address[​](#requesting-a-specific-ip-address) You can also request a specific IP address through [Kubernetes annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) with Calico Cloud IPAM. There are two annotations to request a specific IP address: - `cni.projectcalico.org/ipAddrs`: A list of IPv4 and/or IPv6 addresses to assign to the Pod. The requested IP addresses will be assigned from Calico Cloud IPAM and must exist within a configured IP pool. Example: ```yaml annotations: 'cni.projectcalico.org/ipAddrs': '["192.168.0.1"]' ``` - `cni.projectcalico.org/ipAddrsNoIpam`: A list of IPv4 and/or IPv6 addresses to assign to the Pod, bypassing IPAM. Any IP conflicts and routing have to be taken care of manually or by some other system. Calico Cloud will only distribute routes to a Pod if its IP address falls within a Calico Cloud IP pool using BGP mode. Calico will not distribute ipAddrsNoIpam routes when operating in VXLAN mode. If you assign an IP address that is not in a Calico Cloud IP pool or if its IP address falls within a Calico Cloud IP pool that uses VXLAN encapsulation, you must ensure that routing to that IP address is taken care of through another mechanism. Example: ```yaml annotations: 'cni.projectcalico.org/ipAddrsNoIpam': '["10.0.0.1"]' ``` The ipAddrsNoIpam feature is disabled by default. It can be enabled in the feature\_control section of the CNI network config: ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "ipam": { "type": "calico-ipam" }, "feature_control": { "ip_addrs_no_ipam": true } } ``` > **WARNING:** This feature allows for the bypassing of network policy via IP spoofing. Users should make sure the proper admission control is in place to prevent users from selecting arbitrary IP addresses. > **SECONDARY:** > > - The `ipAddrs` and `ipAddrsNoIpam` annotations can't be used together. > - You can only specify one IPv4/IPv6 or one IPv4 and one IPv6 address with these annotations. > - When `ipAddrs` or `ipAddrsNoIpam` is used with `ipv4pools` or `ipv6pools`, `ipAddrs` / `ipAddrsNoIpam` take priority. #### Requesting a floating IP[​](#requesting-a-floating-ip) You can request a floating IP address for a pod through [Kubernetes annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) with Calico Cloud. > **SECONDARY:** The specified address must belong to an IP Pool for advertisement to work properly. - `cni.projectcalico.org/floatingIPs`: A list of floating IPs which will be assigned to the pod's workload endpoint. Example: ```yaml annotations: 'cni.projectcalico.org/floatingIPs': '["10.0.0.1"]' ``` The floatingIPs feature is disabled by default. It can be enabled in the feature\_control section of the CNI network config: ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "ipam": { "type": "calico-ipam" }, "feature_control": { "floating_ips": true } } ``` > **WARNING:** This feature can allow pods to receive traffic which may not have been intended for that pod. Users should make sure the proper admission control is in place to prevent users from selecting arbitrary floating IP addresses. ### Using IP pools node selectors[​](#using-ip-pools-node-selectors) Nodes will only assign workload addresses from IP pools which select them. By default, IP pools select all nodes, but this can be configured using the `nodeSelector` field. Check out the [IP pool resource document](https://docs.tigera.io/calico-cloud/reference/resources/ippool) for more details. Example: 1. Create (or update) an IP pool that only allocates IPs for nodes where it contains a label `rack=0`. ```bash kubectl create -f -< **SECONDARY:** Be aware of the following [portmap plugin CNI issue](https://github.com/containernetworking/cni/issues/605) where draining nodes may take a long time with a cluster of 100+ nodes and 4000+ services. To disable it, remove the portmap section from the CNI network configuration in the Calico Cloud manifests. ```json { "type": "portmap", "snat": true, "capabilities": { "portMappings": true } } ``` **Traffic shaping plugin** The [traffic shaping Kubernetes CNI plugin](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) supports pod ingress and egress traffic shaping. This bandwidth management technique delays the flow of certain types of network packets to ensure network performance for higher priority applications. It is enabled by default. You can add the `kubernetes.io/ingress-bandwidth` and `kubernetes.io/egress-bandwidth` annotations to your pod. For example, the following sets a 1 megabit-per-second connection for ingress and egress traffic. ```bash apiVersion: v1 kind: Pod metadata: annotations: kubernetes.io/ingress-bandwidth: 1M kubernetes.io/egress-bandwidth: 1M ... ``` To disable it, remove the bandwidth section from the CNI network configuration in the Calico Cloud manifests. ```json { "type": "bandwidth", "capabilities": { "bandwidth": true } } ``` ### Order of precedence[​](#order-of-precedence) If more than one of these methods are used for IP address assignment, they will take on the following precedence, 1 being the highest: 1. Kubernetes annotations 2. CNI configuration 3. IP pool node selectors > **SECONDARY:** Calico Cloud IPAM will not reassign IP addresses to workloads that are already running. To update running workloads with IP addresses from a newly configured IP pool, they must be recreated. We recommend doing this before going into production or during a maintenance window. ### Specify num\_queues for veth interfaces[​](#specify-num_queues-for-veth-interfaces) `num_rx_queues` and `num_tx_queues` can be set using the `num_queues` option in the CNI configuration. Default: 1 For example: ```json { "num_queues": 3 } ``` **Tab: Tab 3** The `host-local` IPAM plugin can be configured by setting the `Spec.CNI.IPAM.Plugin` field to `HostLocal` on the [operator.tigera.io/Installation](https://docs.tigera.io/calico-cloud/reference/installation/api#installation) API. Calico will use the `host-local` IPAM plugin to allocate IPv4 addresses from the node's IPv4 pod CIDR if there is an IPv4 pool configured in `Spec.IPPools`, and an IPv6 address from the node's IPv6 pod CIDR if there is an IPv6 pool configured in `Spec.IPPools`. The following example configures Calico to assign dual-stack IPs to pods using the host-local IPAM plugin. ```yaml kind: Installation apiVersion: operator.tigera.io/v1 metadata: name: default spec: calicoNetwork: ipPools: - cidr: 192.168.0.0/16 - cidr: 2001:db8::/64 cni: type: Calico ipam: type: HostLocal ``` **Tab: Tab 4** When using the CNI `host-local` IPAM plugin, two special values - `usePodCidr` and `usePodCidrIPv6` - are allowed for the subnet field (either at the top-level, or in a "range"). This tells the plugin to determine the subnet to use from the Kubernetes API based on the Node.podCIDR field. Calico Cloud does not use the `gateway` field of a range so that field is not required and it will be ignored if present. > **SECONDARY:** `usePodCidr` and `usePodCidrIPv6` can only be used as the value of the `subnet` field, it cannot be used in `rangeStart` or `rangeEnd` so those values are not useful if `subnet` is set to `usePodCidr`. Calico Cloud supports the host-local IPAM plugin's `routes` field as follows: - If there is no `routes` field, Calico Cloud will install a default `0.0.0.0/0`, and/or `::/0` route into the pod (depending on whether the pod has an IPv4 and/or IPv6 address). - If there is a `routes` field then Calico Cloud will program *only* the routes in the routes field into the pod. Since Calico Cloud implements a point-to-point link into the pod, the `gw` field is not required and it will be ignored if present. All routes that Calico Cloud installs will have Calico Cloud's link-local IP as the next hop. Calico Cloud CNI plugin configuration: - `node_name` - The node name to use when looking up the CIDR value (defaults to current hostname) ```json { "name": "any_name", "cniVersion": "0.1.0", "type": "calico", "kubernetes": { "kubeconfig": "/path/to/kubeconfig", "node_name": "node-name-in-k8s" }, "ipam": { "type": "host-local", "ranges": [[{ "subnet": "usePodCidr" }], [{ "subnet": "usePodCidrIPv6" }]], "routes": [{ "dst": "0.0.0.0/0" }, { "dst": "2001:db8::/96" }] } } ``` When making use of the `usePodCidr` or `usePodCidrIPv6` options, the Calico Cloud CNI plugin requires read-only Kubernetes API access to the `Nodes` resource. #### Configuring node and typha[​](#configuring-node-and-typha) When using `host-local` IPAM with the Kubernetes API datastore, you must configure both cnx-node and the Typha deployment to use the `Node.podCIDR` field by setting the environment variable `USE_POD_CIDR=true` in each. ### Configure resource requests and limits ## Big picture[​](#big-picture) Resource requests and limits are essential configurations for managing resource allocation and ensuring optimal performance of Kubernetes workloads. In Calico Cloud, these configurations can be customized using custom resources to meet specific requirements and optimize resource utilization. > **SECONDARY:** It's important to note that the CPU and memory values used in the examples are for demonstration purposes and should be adjusted based on individual system requirements. To find the list of all applicable containers for a component, please refer to its specification. ## APIServer custom resource[​](#apiserver-custom-resource) The [APIServer](https://docs.tigera.io/calico-cloud/reference/installation/api#apiserver) CR provides a way to configure APIServerDeployment. The following sections provide example configurations for this CR. ### APIServerDeployment[​](#apiserverdeployment) To configure resource specification for the [APIServerDeployment](https://docs.tigera.io/calico-cloud/reference/installation/api#apiserverdeployment), patch the installation CR using the below command: ```bash $ kubectl patch apiserver tigera-secure --type=merge --patch='{"spec": {"apiServerDeployment":{"spec": {"template": {"spec": {"containers":[{"name":"calico-apiserver","resources":{"limits":{"cpu":"1", "memory":"1000Mi"},"requests":{"cpu":"100m", "memory":"100Mi"}}},{"name":"tigera-queryserver","resources":{"limits":{"cpu":"1", "memory":"1000Mi"},"requests":{"cpu":"100m", "memory":"100Mi"}}}]}}}}}}' ``` This command sets the CPU request to 100 milliCPU (mCPU) and the memory request is set to 100 Mebibytes (MiB) while the CPU limit is set to 1 CPU and the memory limit is set to 1000 Mebibytes (MiB). #### Verification[​](#verification) You can verify the configured resources using the following command: ```bash $ kubectl get deployment.apps/calico-apiserver -n calico-system -o json | jq '.spec.template.spec.containers[] | {name: .name, resources: .resources}' ``` This command will output the configured resource requests and limits for the Calico APIServerDeployment component in JSON format. ```bash { "name": "calico-apiserver", "resources": { "limits": { "cpu": "1", "memory": "1000Mi" }, "requests": { "cpu": "100m", "memory": "100Mi" } } } { "name": "tigera-queryserver", "resources": { "limits": { "cpu": "1", "memory": "1000Mi" }, "requests": { "cpu": "100m", "memory": "100Mi" } } } ``` ## ApplicationLayer custom resource[​](#applicationlayer-custom-resource) The [ApplicationLayer](https://docs.tigera.io/calico-cloud/reference/installation/api#applicationlayer) CR provides a way to configure resources for L7LogCollectorDaemonSet. The following sections provide example configurations for this CR. ### L7LogCollectorDaemonSet[​](#l7logcollectordaemonset) To configure resource specification for the [L7LogCollectorDaemonSet](https://docs.tigera.io/calico-cloud/reference/installation/api#l7logcollectordaemonset), patch the ApplicationLayer CR using the below command: ```bash $ kubectl patch applicationlayer tigera-secure --type=merge --patch='{"spec": {"l7LogCollectorDaemonSet":{"spec": {"template": {"spec": {"containers":[{"name":"l7-collector","resources":{"limits":{"cpu":"1", "memory":"1000Mi"},"requests":{"cpu":"100m", "memory":"100Mi"}}},{"name":"envoy-proxy","resources":{"limits":{"cpu":"1", "memory":"1000Mi"},"requests":{"cpu":"100m", "memory":"100Mi"}}}]}}}}}}' applicationlayer.operator.tigera.io/tigera-secure patched ``` This command sets the CPU request to 100 milliCPU (mCPU) and the memory request is set to 100 Mebibytes (MiB) while the CPU limit is set to 1 CPU and the memory limit is set to 1000 Mebibytes (MiB). #### Verification[​](#verification-1) You can verify the configured resources using the following command: ```bash $ kubectl get daemonset.apps/l7-log-collector -n calico-system -o json | jq '.spec.template.spec.containers[] | {name: .name, resources: .resources}' ``` This command will output the configured resource requests and limits for the Calico L7LogCollectorDaemonSet component in JSON format. ```bash { "name": "envoy-proxy", "resources": { "limits": { "cpu": "1", "memory": "1000Mi" }, "requests": { "cpu": "100m", "memory": "100Mi" } } } { "name": "l7-collector", "resources": { "limits": { "cpu": "1", "memory": "1000Mi" }, "requests": { "cpu": "100m", "memory": "100Mi" } } } ``` ## Compliance custom resource[​](#compliance-custom-resource) The [Compliance](https://docs.tigera.io/calico-cloud/reference/installation/api#compliance) CR provides a way to configure resources for ComplianceControllerDeployment, ComplianceSnapshotterDeployment, ComplianceBenchmarkerDaemonSet, ComplianceServerDeployment, ComplianceReporterPodTemplate. The following sections provide example configurations for this CR. ### ComplianceControllerDeployment[​](#compliancecontrollerdeployment) To configure resource specification for the [ComplianceControllerDeployment](https://docs.tigera.io/calico-cloud/reference/installation/api#compliancecontrollerdeployment), patch the Compliance CR using the below command: ```bash kubectl patch compliance tigera-secure --type=merge --patch='{"spec": {"complianceControllerDeployment":{"spec": {"template": {"spec": {"containers":[{"name":"compliance-controller","resources":{"limits":{"cpu":"1", "memory":"1000Mi"},"requests":{"cpu":"100m", "memory":"100Mi"}}}]}}}}}}' ``` This command sets the CPU request to 100 milliCPU (mCPU) and the memory request is set to 100 Mebibytes (MiB) while the CPU limit is set to 1 CPU and the memory limit is set to 1000 Mebibytes (MiB). #### Verification[​](#verification-2) You can verify the configured resources using the following command: ```bash kubectl get deployment.apps/compliance-controller -n tigera-compliance -o json|jq '.spec.template.spec.containers[] | {name: .name, resources: .resources}' ``` This command will output the configured resource requests and limits for the ComplianceControllerDeployment component in JSON format. ```bash { "name": "compliance-controller", "resources": { "limits": { "cpu": "1", "memory": "1000Mi" }, "requests": { "cpu": "100m", "memory": "100Mi" } } } ``` ### ComplianceSnapshotterDeployment[​](#compliancesnapshotterdeployment) To configure resource specification for the [ComplianceSnapshotterDeployment](https://docs.tigera.io/calico-cloud/reference/installation/api#compliancesnapshotterdeployment), patch the Compliance CR using the below command: ```bash kubectl patch compliance tigera-secure --type=merge --patch='{"spec": {"complianceSnapshotterDeployment":{"spec": {"template": {"spec": {"containers":[{"name":"compliance-snapshotter","resources":{"limits":{"cpu":"1", "memory":"1000Mi"},"requests":{"cpu":"100m", "memory":"100Mi"}}}]}}}}}}' ``` This command sets the CPU request to 100 milliCPU (mCPU) and the memory request is set to 100 Mebibytes (MiB) while the CPU limit is set to 1 CPU and the memory limit is set to 1000 Mebibytes (MiB). #### Verification[​](#verification-3) You can verify the configured resources using the following command: ```bash kubectl get deployment.apps/compliance-snapshotter -n tigera-compliance -o json|jq '.spec.template.spec.containers[] | {name: .name, resources: .resources}' ``` This command will output the configured resource requests and limits for the ComplianceSnapshotterDeployment in JSON format. ```bash { "name": "compliance-snapshotter", "resources": { "limits": { "cpu": "1", "memory": "1000Mi" }, "requests": { "cpu": "100m", "memory": "100Mi" } } } ``` ### ComplianceBenchmarkerDaemonSet[​](#compliancebenchmarkerdaemonset) To configure resource specification for the [ComplianceBenchmarkerDaemonSet](https://docs.tigera.io/calico-cloud/reference/installation/api#compliancebenchmarkerdaemonset), patch the Compliance CR using the below command: ```bash kubectl patch compliance tigera-secure --type=merge --patch='{"spec": {"complianceBenchmarkerDaemonSet":{"spec": {"template": {"spec": {"containers":[{"name":"compliance-benchmarker","resources":{"limits":{"cpu":"1", "memory":"1000Mi"},"requests":{"cpu":"100m", "memory":"100Mi"}}}]}}}}}}' ``` This command sets the CPU request to 100 milliCPU (mCPU) and the memory request is set to 100 Mebibytes (MiB) while the CPU limit is set to 1 CPU and the memory limit is set to 1000 Mebibytes (MiB). #### Verification[​](#verification-4) You can verify the configured resources using the following command: ```bash kubectl get daemonset.apps/compliance-benchmarker -n tigera-compliance -o json |jq '.spec.template.spec.containers[] | {name: .name, resources: .resources}' ``` ```bash { "name": "compliance-benchmarker", "resources": { "limits": { "cpu": "1", "memory": "1000Mi" }, "requests": { "cpu": "100m", "memory": "100Mi" } } } ``` This command will output the configured resource requests and limits for the ComplianceBenchmarkerDaemonSet in JSON format. ### ComplianceServerDeployment[​](#complianceserverdeployment) To configure resource specification for the [ComplianceServerDeployment](https://docs.tigera.io/calico-cloud/reference/installation/api#complianceserverdeployment), patch the Compliance CR using the below command: ```bash kubectl patch compliance tigera-secure --type=merge --patch='{"spec": {"complianceServerDeployment":{"spec": {"template": {"spec": {"containers":[{"name":"compliance-server","resources":{"limits":{"cpu":"1", "memory":"1000Mi"},"requests":{"cpu":"100m", "memory":"100Mi"}}}]}}}}}}' ``` This command sets the CPU request to 100 milliCPU (mCPU) and the memory request is set to 100 Mebibytes (MiB) while the CPU limit is set to 1 CPU and the memory limit is set to 1000 Mebibytes (MiB). #### Verification[​](#verification-5) You can verify the configured resources using the following command: ```bash kubectl get deployment.apps/compliance-server -n tigera-compliance -o json| jq '.spec.template.spec.containers[] | {name: .name, resources: .resources}' ``` This command will output the configured resource requests and limits for the ComplianceServerDeployment in JSON format. ```bash { "name": "compliance-server", "resources": { "limits": { "cpu": "1", "memory": "1000Mi" }, "requests": { "cpu": "100m", "memory": "100Mi" } } } ``` ### ComplianceReporterPodTemplate.[​](#compliancereporterpodtemplate) To configure resource specification for the [ComplianceReporterPodTemplate](https://docs.tigera.io/calico-cloud/reference/installation/api#compliancereporterpodtemplate), patch the Compliance CR using the below command: ```bash kubectl patch compliance tigera-secure --type=merge --patch='{"spec": {"complianceReporterPodTemplate": {"template": {"spec": {"containers":[{"name":"reporter","resources":{"limits":{"cpu":"1", "memory":"1000Mi"},"requests":{"cpu":"100m", "memory":"100Mi"}}}]}}}}}' ``` This command sets the CPU request to 100 milliCPU (mCPU) and the memory request is set to 100 Mebibytes (MiB) while the CPU limit is set to 1 CPU and the memory limit is set to 1000 Mebibytes (MiB). #### Verification[​](#verification-6) You can verify the configured resources using the following command: ```bash kubectl get Podtemplates tigera.io.report -n tigera-compliance -o json | jq '.template.spec.containers[] | {name: .name, resources: .resources}' ``` This command will output the configured resource requests and limits for the ComplianceReporterPodTemplate component in JSON format. ```bash { "name": "reporter", "resources": { "limits": { "cpu": "1", "memory": "1000Mi" }, "requests": { "cpu": "100m", "memory": "100Mi" } } } ``` ## Installation custom resource[​](#installation-custom-resource) The [Installation CR](https://docs.tigera.io/calico-cloud/reference/installation/api) provides a way to configure resources for various Calico Enterprise components, including TyphaDeployment, calicoNodeDaemonSet, CalicoNodeWindowsDaemonSet, csiNodeDriverDaemonSet and KubeControllersDeployment. The following sections provide example configurations for this CR. Example Configurations: ### TyphaDeployment[​](#typhadeployment) To configure resource specification for the [TyphaDeployment](https://docs.tigera.io/calico-cloud/reference/installation/api#typhadeployment), patch the installation CR using the below command: ```bash kubectl patch installations default --type=merge --patch='{"spec": {"typhaDeployment": {"spec": {"template": {"spec": {"containers": [{"name": "calico-typha", "resources": {"requests": {"cpu": "100m", "memory": "100Mi"}, "limits": {"cpu": "1", "memory": "1000Mi"}}}]}}}}}}' ``` This command sets the CPU request to 100 milliCPU (mCPU) and the memory request is set to 100 Mebibytes (MiB) while the CPU limit is set to 1 CPU and the memory limit is set to 1000 Mebibytes (MiB). #### Verification[​](#verification-7) You can verify the configured resources using the following command: ```bash $ kubectl get deployment.apps/calico-typha -n calico-system -o json | jq '.spec.template.spec.containers[]| {name:.name,resources:.resources}' ``` This command will output the configured resource requests and limits for the Calico TyphaDeployment component in JSON format. ```bash { "name": "calico-typha", "resources": { "limits": { "cpu": "1", "memory": "1000Mi" }, "requests": { "cpu": "100m", "memory": "100Mi" } } } ``` ### CalicoNodeDaemonSet[​](#caliconodedaemonset) To configure resource requests for the [calicoNodeDaemonSet](https://docs.tigera.io/calico-cloud/reference/installation/api#caliconodedaemonset) component, patch the installation CR using the below command: ```bash $ kubectl patch installations default --type=merge --patch='{"spec": {"calicoNodeDaemonSet":{"spec": {"template": {"spec": {"containers":[{"name":"calico-node","resources":{"requests":{"cpu":"100m", "memory":"100Mi"}, "limits":{"cpu":"1", "memory":"1000Mi"}}}]}}}}}}' ``` This command sets the CPU request to 100 milliCPU (mCPU) and the memory request is set to 100 Mebibytes (MiB) while the CPU limit is set to 1 CPU and the memory limit is set to 1000 Mebibytes (MiB). #### Verification[​](#verification-8) You can verify the configured resources using the following command: ```bash $ kubectl get daemonset.apps/calico-node -n calico-system -o json | jq '.spec.template.spec.containers[]| {name:.name,resources:.resources}' ``` This command will output the configured resource requests and limits for the Calico calicoNodeDaemonSet component in JSON format. ```bash { "name": "calico-node", "resources": { "limits": { "cpu": "1", "memory": "1000Mi" }, "requests": { "cpu": "100m", "memory": "100Mi" } } } ``` ### calicoNodeWindowsDaemonSet[​](#caliconodewindowsdaemonset) To configure resource requests for the [calicoNodeWindowsDaemonSet](https://docs.tigera.io/calico-cloud/reference/installation/api#caliconodewindowsdaemonset) component, patch the installation CR using the below command: ```bash $ kubectl patch installations default --type=merge --patch='{"spec": {"calicoNodeWindowsDaemonSet":{"spec": {"template": {"spec": {"containers":[{"name":"calico-node-windows","resources":{"requests":{"cpu":"100m", "memory":"100Mi"}, "limits":{"cpu":"1", "memory":"1000Mi"}}}]}}}}}}' ``` This command sets the CPU request to 100 milliCPU (mCPU) and the memory request is set to 100 Mebibytes (MiB) while the CPU limit is set to 1 CPU and the memory limit is set to 1000 Mebibytes (MiB). #### Verification[​](#verification-9) You can verify the configured resources using the following command: ```bash $ kubectl get daemonset.apps/calico-node -n calico-system -o json | jq '.spec.template.spec.containers[]| {name:.name,resources:.resources}' ``` This command will output the configured resource requests and limits for the Calico calicoNodeWindowsDaemonSet component in JSON format. ```bash { "name": "calico-node", "resources": { "limits": { "cpu": "1", "memory": "1000Mi" }, "requests": { "cpu": "100m", "memory": "100Mi" } } } ``` ### CalicoKubeControllersDeployment[​](#calicokubecontrollersdeployment) To configure resource requests for the [CalicoKubeControllersDeployment](https://docs.tigera.io/calico-cloud/reference/installation/api#calicokubecontrollersdeployment) component, patch the installation CR using the below command: ```bash $ kubectl patch installations default --type=merge --patch='{"spec": {"calicoKubeControllersDeployment":{"spec": {"template": {"spec": {"containers":[{"name":"calico-kube-controllers","resources":{"requests":{"cpu":"100m", "memory":"100Mi"}, "limits":{"cpu":"1", "memory":"1000Mi"}}}]}}}}}}' ``` This command sets the CPU request to 100 milliCPU (mCPU) and the memory request is set to 100 Mebibytes (MiB) while the CPU limit is set to 1 CPU and the memory limit is set to 1000 Mebibytes (MiB). #### Verification[​](#verification-10) You can verify the configured resources using the following command: ```bash $ kubectl get deployment.apps/calico-kube-controllers -n calico-system -o json | jq '.spec.template.spec.containers[]| {name:.name,resources:.resources}' ``` This command will output the configured resource requests and limits for the Calico CalicoKubeControllersDeployment component in JSON format. ```bash { "name": "calico-kube-controllers", "resources": { "limits": { "cpu": "1", "memory": "1000Mi" }, "requests": { "cpu": "100m", "memory": "100Mi" } } } ``` ### CSINodeDriverDaemonSet[​](#csinodedriverdaemonset) To configure resource requests for the [CSINodeDriverDaemonSet](https://docs.tigera.io/calico-cloud/reference/installation/api#csinodedriverdaemonset) component, patch the installation CR using the below command: ```bash $ kubectl patch installations default --type=merge --patch='{"spec": {"csiNodeDriverDaemonSet":{"spec": {"template": {"spec": {"containers":[{"name":"calico-csi","resources":{"requests":{"cpu":"100m", "memory":"100Mi"}, "limits":{"cpu":"1", "memory":"1000Mi"}}},{"name":"csi-node-driver-registrar","resources":{"requests":{"cpu":"50m", "memory":"50Mi"}, "limits":{"cpu":"1", "memory":"1000Mi"}}}]}}}}}}' ``` This command sets the CPU request to 100 milliCPU (mCPU) and the memory request is set to 100 Mebibytes (MiB) while the CPU limit is set to 1 CPU and the memory limit is set to 1000 Mebibytes (MiB). #### Verification[​](#verification-11) You can verify the configured resources using the following command: ```bash $ kubectl get daemonset.apps/csi-node-driver -n calico-system -o json | jq '.spec.template.spec.containers[]| {name:.name,resources:.resources}' ``` This command will output the configured resource requests and limits for the Calico calicoNodeDaemonSet component in JSON format. ```bash { "name": "calico-csi", "resources": { "limits": { "cpu": "1", "memory": "1000Mi" }, "requests": { "cpu": "100m", "memory": "100Mi" } } } { "name": "csi-node-driver-registrar", "resources": { "limits": { "cpu": "1", "memory": "1000Mi" }, "requests": { "cpu": "50m", "memory": "50Mi" } } } ``` ## IntrusionDetection custom resource[​](#intrusiondetection-custom-resource) The [IntrusionDetection](https://docs.tigera.io/calico-cloud/reference/installation/api#intrusiondetection) CR provides a way to configure resources for IntrusionDetectionControllerDeployment. The following sections provide example configurations for this CR. ### IntrusionDetectionControllerDeployment.[​](#intrusiondetectioncontrollerdeployment) To configure resource specification for the [IntrusionDetectionControllerDeployment](https://docs.tigera.io/calico-cloud/reference/installation/api#intrusiondetectioncontrollerdeployment), patch the IntrusionDetection CR using the below command: ```bash $ kubectl patch intrusiondetection tigera-secure --type=merge --patch='{"spec": {"intrusionDetectionControllerDeployment":{"spec": {"template": {"spec": {"containers":[{"name":"webhooks-processor","resources":{"limits":{"cpu":"1", "memory":"1000Mi"},"requests":{"cpu":"100m", "memory":"1000Mi"}}},{"name":"controller","resources":{"limits":{"cpu":"1", "memory":"1000Mi"},"requests":{"cpu":"100m", "memory":"1000Mi"}}}]}}}}}}' ``` This command sets the CPU request to 100 milliCPU (mCPU) and the memory request is set to 100 Mebibytes (MiB) while the CPU limit is set to 1 CPU and the memory limit is set to 1000 Mebibytes (MiB). #### Verification[​](#verification-12) You can verify the configured resources using the following command: ```bash $ kubectl get deployment.apps/intrusion-detection-controller -n tigera-intrusion-detection -o json|jq '.spec.template.spec.containers[] | {name: .name, resources: .resources}' ``` This command will output the configured resource requests and limits for the IntrusionDetectionControllerDeployment in JSON format. ```bash { "name": "controller", "resources": { "limits": { "cpu": "1", "memory": "1000Mi" }, "requests": { "cpu": "100m", "memory": "1000Mi" } } } { "name": "webhooks-processor", "resources": { "limits": { "cpu": "1", "memory": "1000Mi" }, "requests": { "cpu": "100m", "memory": "1000Mi" } } } ``` ## LogCollector custom resource[​](#logcollector-custom-resource) The [LogCollector](https://docs.tigera.io/calico-cloud/reference/installation/api#logcollector) CR provides a way to configure resources for FluentdDaemonSet, EKSLogForwarderDeployment. ### FluentdDaemonSet.[​](#fluentddaemonset) To configure resource specification for the [FluentdDaemonSet](https://docs.tigera.io/calico-cloud/reference/installation/api#fluentddaemonset), patch the LogCollector CR using the below command: ```bash kubectl patch logcollector tigera-secure --type=merge --patch='{"spec": {"fluentdDaemonSet":{"spec": {"template": {"spec": {"containers":[{"name":"fluentd","resources":{"limits":{"cpu":"1", "memory":"1000Mi"},"requests":{"cpu":"100m", "memory":"100Mi"}}}]}}}}}}' ``` This command sets the CPU request to 100 milliCPU (mCPU) and the memory request is set to 100 Mebibytes (MiB) while the CPU limit is set to 1 CPU and the memory limit is set to 1000 Mebibytes (MiB). #### Verification[​](#verification-13) You can verify the configured resources using the following command: ```bash kubectl get daemonset.apps/fluentd-node -n tigera-fluentd -o json | jq '.spec.template.spec.containers[] | {name: .name, resources: .resources}' ``` This command will output the configured resource requests and limits for the FluentdDaemonSet in JSON format. ```bash { "name": "fluentd", "resources": { "limits": { "cpu": "1", "memory": "1000Mi" }, "requests": { "cpu": "100m", "memory": "100Mi" } } } ``` ### EKSLogForwarderDeployment.[​](#ekslogforwarderdeployment) To configure resource specification for the [EKSLogForwarderDeployment](https://docs.tigera.io/calico-cloud/reference/installation/api#ekslogforwarderdeployment), patch the LogCollector CR using the below command: ```bash kubectl patch logcollector tigera-secure --type=merge --patch='{"spec": {"eksLogForwarderDeployment": {"spec": {"template": {"spec": {"containers":[{"name":"eks-log-forwarder","resources":{"limits":{"cpu":"1", "memory":"1000Mi"},"requests":{"cpu":"100m", "memory":"100Mi"}}}]}}}}}}' ``` This command sets the CPU request to 100 milliCPU (mCPU) and the memory request is set to 100 Mebibytes (MiB) while the CPU limit is set to 1 CPU and the memory limit is set to 1000 Mebibytes (MiB). #### Verification[​](#verification-14) You can verify the configured resources using the following command: ```bash kubectl get deployment.apps/eks-log-forwarder -n tigera-fluentd -o json | jq '.spec.template.spec.containers[] | {name: .name, resources: .resources}' ``` This command will output the configured resource requests and limits for the EKSLogForwarderDeployment in JSON format. ```bash { "name": "eks-log-forwarder", "resources": { "limits": { "cpu": "1", "memory": "1000Mi" }, "requests": { "cpu": "100m", "memory": "100Mi" } } } ``` ## ManagementClusterConnection custom resource[​](#managementclusterconnection-custom-resource) The [ManagementClusterConnection](https://docs.tigera.io/calico-cloud/reference/installation/api#managementclusterconnection) CR provides a way to configure resources for GuardianDeployment. The following sections provide example configurations for this CR. ### GuardianDeployment.[​](#guardiandeployment) To configure resource specification for the [GuardianDeployment](https://docs.tigera.io/calico-cloud/reference/installation/api#guardiandeployment), patch the ManagementClusterConnection CR using the below command: ```bash kubectl patch managementclusterconnection tigera-secure --type=merge --patch='{"spec": {"guardianDeployment":{"spec": {"template": {"spec": {"containers":[{"name":"tigera-guardian","resources":{"limits":{"cpu":"1", "memory":"1000Mi"},"requests":{"cpu":"100m", "memory":"100Mi"}}}]}}}}}}' ``` This command sets the CPU request to 100 milliCPU (mCPU) and the memory request is set to 100 Mebibytes (MiB) while the CPU limit is set to 1 CPU and the memory limit is set to 1000 Mebibytes (MiB). #### Verification[​](#verification-15) You can verify the configured resources using the following command: ```bash kubectl get deployment.apps/tigera-guardian -n tigera-guardian -o json | jq '.spec.template.spec.containers[] | {name: .name, resources: .resources}' ``` This command will output the configured resource requests and limits for the GuardianDeployment in JSON format. ```bash { "limits": { "cpu": "1", "memory": "1000Mi" }, "requests": { "cpu": "100m", "memory": "100Mi" } } ``` ## PacketCaptureAPI custom resource[​](#packetcaptureapi-custom-resource) The [PacketCaptureAPI](https://docs.tigera.io/calico-cloud/reference/installation/api#packetcaptureapi) CR provides a way to configure resources for PacketCapture. The following sections provide example configurations for this CR. ### PacketCaptureAPIDeployment[​](#packetcaptureapideployment) To configure resource specification for the [PacketCaptureAPI](https://docs.tigera.io/calico-cloud/reference/installation/api#packetcaptureapi), patch the PacketCapture CR using the below command: ```bash kubectl patch packetcaptureapis tigera-secure --type=merge --patch='{"spec": {"packetCaptureAPIDeployment":{"spec": {"template": {"spec": {"containers":[{"name":"tigera-packetcapture-server","resources":{"limits":{"cpu":"1", "memory":"1000Mi"},"requests":{"cpu":"100m", "memory":"100Mi"}}}]}}}}}}' ``` This command sets the CPU request to 100 milliCPU (mCPU) and the memory request is set to 100 Mebibytes (MiB) while the CPU limit is set to 1 CPU and the memory limit is set to 1000 Mebibytes (MiB). #### Verification[​](#verification-16) You can verify the configured resources using the following command: ```bash kubectl get deployment.apps/tigera-packetcapture -n tigera-packetcapture -o json | jq '.spec.template.spec.containers[] | {name: .name, resources: .resources}' ``` This command will output the configured resource requests and limits for the PacketCaptureDeployment in JSON format. ```bash { "name": "tigera-packetcapture-server", "resources": { "limits": { "cpu": "1", "memory": "1000Mi" }, "requests": { "cpu": "100m", "memory": "100Mi" } } } ``` ### kube-controllers ## [📄️ Configuring the Calico Cloud Kubernetes controllers](https://docs.tigera.io/calico-cloud/reference/component-resources/kube-controllers/configuration) [Calico Cloud Kubernetes controllers monitor the Kubernetes API and perform actions based on cluster state.](https://docs.tigera.io/calico-cloud/reference/component-resources/kube-controllers/configuration) ## [📄️ Monitoring kube-controllers with Prometheus](https://docs.tigera.io/calico-cloud/reference/component-resources/kube-controllers/prometheus) [Review metrics for the kube-controllers component if you are using Prometheus.](https://docs.tigera.io/calico-cloud/reference/component-resources/kube-controllers/prometheus) ### Configuring the Calico Cloud Kubernetes controllers The Calico Cloud Kubernetes controllers are deployed in a Kubernetes cluster. The different controllers monitor the Kubernetes API and perform actions based on cluster state. **Tab: Operator** If you have installed Calico using the operator, see the [KubeControllersConfiguration](https://docs.tigera.io/calico-cloud/reference/resources/kubecontrollersconfig) resource instead. **Tab: Manifest** The controllers are primarily configured through environment variables. When running the controllers as a Kubernetes pod, this is accomplished through the pod manifest `env` section. ## The tigera/kube-controllers container[​](#the-tigerakube-controllers-container) The `tigera/kube-controllers` container includes the following controllers: 1. node controller: watches for the removal of Kubernetes nodes and removes corresponding data from Calico Cloud, and optionally watches for node updates to create and sync host endpoints for each node. 2. federation controller: watches Kubernetes services and endpoints locally and across all remote clusters, and programs Kubernetes endpoints for any locally configured service that specifies a service federation selector annotation. ### Configuring datastore access[​](#configuring-datastore-access) The datastore type can be configured via the `DATASTORE_TYPE` environment variable. Only supported value is `kubernetes`. #### kubernetes[​](#kubernetes) When running the controllers as a Kubernetes pod, Kubernetes API access is [configured automatically](https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#accessing-the-api-from-a-pod) and no additional configuration is required. However, the controllers can also be configured to use an explicit [kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) file override to configure API access if needed. | Environment | Description | Schema | | ------------ | ------------------------------------------------------------------ | ------ | | `KUBECONFIG` | Path to a Kubernetes kubeconfig file mounted within the container. | path | ### Other configuration[​](#other-configuration) > **SECONDARY:** Whenever possible, prefer configuring the kube-controllers component using the [KubeControllersConfiguration](https://docs.tigera.io/calico-cloud/reference/resources/kubecontrollersconfig) API resource, Some configuration options may not be available through environment variables. The following environment variables can be used to configure the Calico Cloud Kubernetes controllers. | Environment | Description | Schema | Default | | --------------------- | --------------------------------------------------------------------------- | --------------------------------------------------------- | ----------------------------------------------------- | | `DATASTORE_TYPE` | Which datastore type to use | etcdv3, kubernetes | kubernetes | | `ENABLED_CONTROLLERS` | Which controllers to run | namespace, node, policy, serviceaccount, workloadendpoint | policy,namespace,serviceaccount,workloadendpoint,node | | `LOG_LEVEL` | Minimum log level to be displayed. | debug, info, warning, error | info | | `KUBECONFIG` | Path to a kubeconfig file for Kubernetes API access | path | | | `SYNC_NODE_LABELS` | When enabled, Kubernetes node labels will be copied to Calico node objects. | boolean | true | | `AUTO_HOST_ENDPOINTS` | When set to enabled, automatically create a host endpoint for each node. | enabled, disabled | disabled | ## About each controller[​](#about-each-controller) ### Node controller[​](#node-controller) The node controller has several functions. - Garbage collects IP addresses. - Automatically provisions host endpoints for Kubernetes nodes. ### Federation controller[​](#federation-controller) The federation controller syncs Kubernetes federated endpoint changes to the Calico Cloud datastore. The controller must have read access to the Kubernetes API to monitor `Service` and `Endpoints` events, and must also have write access to update `Endpoints`. The federation controller is disabled by default if `ENABLED_CONTROLLERS` is not explicitly specified. This controller is valid for all Calico Cloud datastore types. For more details refer to the [Configuring federated services](https://docs.tigera.io/calico-cloud/multicluster/services-controller) usage guide. ### Monitoring kube-controllers with Prometheus kube-controllers can be configured to report a number of metrics through Prometheus. This reporting is enabled by default on port 9094. See the [configuration reference](https://docs.tigera.io/calico-cloud/reference/resources/kubecontrollersconfig) for how to change metrics reporting configuration (or disable it completely). ## Metric reference[​](#metric-reference) #### kube-controllers specific[​](#kube-controllers-specific) kube-controllers exports a number of Prometheus metrics. The current set is as follows. Since some metrics may be tied to particular implementation choices inside kube-controllers we can't make any hard guarantees that metrics will persist across releases. However, we aim not to make any spurious changes to existing metrics. | Metric Name | Labels | Description | | ------------------------------------ | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ipam_allocations_in_use` | ippool, node | Number of Calico IP allocations currently in use by a workload or interface. | | `ipam_allocations_borrowed` | ippool, node | Number of Calico IP allocations currently in use where the allocation was borrowed from a block affine to another node. | | `ipam_allocations_gc_candidates` | ippool, node | Number of Calico IP allocations currently marked by the GC as potential leaks. This metric returns to zero under normal GC operation. | | `ipam_allocations_gc_reclamations` | ippool, node | Count of Calico IP allocations that have been reclaimed by the GC. Increase of this counter corresponds with a decrease of the candidates gauge under normal operation. | | `ipam_blocks` | ippool, node | Number of IPAM blocks. | | `ipam_ippool_size` | ippool | Number of IP addresses in the IP Pool CIDR. | | `ipam_blocks_per_node` | node | Number of IPAM blocks, indexed by the node to which they have affinity. Prefer `ipam_blocks` for new integrations. | | `ipam_allocations_per_node` | node | Number of Calico IP allocations, indexed by node on which the allocation was made. Prefer `ipam_allocations_in_use` for new integrations. | | `ipam_allocations_borrowed_per_node` | node | Number of Calico IP allocations borrowed from a non-affine block, indexed by node on which the allocation was made. Prefer `ipam_allocations_borrowed` for new integrations. | | `remote_cluster_connection_status` | remote\_cluster\_name | Status of the remote cluster connection in federation. Represented as numeric values 0 (NotConnecting) ,1 (Connecting), 2 (InSync), 3 (ReSyncInProgress), 4 (ConfigChangeRestartRequired), 5 (ConfigInComplete). | Labels can be interpreted as follows: | Label Name | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `node` | For allocation metrics, the node on which the allocation was made. For block metrics, the node for which the block has affinity. If the block has no affinity, value will be `no_affinity`. | | `ippool` | The IP Pool that the IPAM block occupies. If there is no IP Pool which matches the block, value will be `no_ippool`. | | `remote_cluster_name` | Name of the remote cluster in federation. | Prometheus metrics are self-documenting, with metrics turned on, `curl` can be used to list the metrics along with their help text and type information. ```bash curl -s http://localhost:9094/metrics | head ``` #### CPU / memory metrics[​](#cpu--memory-metrics) kube-controllers also exports the default set of metrics that Prometheus makes available. Currently, those include: | Name | Description | | -------------------------------------------- | ------------------------------------------------------------------ | | `go_gc_duration_seconds` | A summary of the GC invocation durations. | | `go_goroutines` | Number of goroutines that currently exist. | | `go_memstats_alloc_bytes` | Number of bytes allocated and still in use. | | `go_memstats_alloc_bytes_total` | Total number of bytes allocated, even if freed. | | `go_memstats_buck_hash_sys_bytes` | Number of bytes used by the profiling bucket hash table. | | `go_memstats_frees_total` | Total number of frees. | | `go_memstats_gc_sys_bytes` | Number of bytes used for garbage collection system metadata. | | `go_memstats_heap_alloc_bytes` | Number of heap bytes allocated and still in use. | | `go_memstats_heap_idle_bytes` | Number of heap bytes waiting to be used. | | `go_memstats_heap_inuse_bytes` | Number of heap bytes that are in use. | | `go_memstats_heap_objects` | Number of allocated objects. | | `go_memstats_heap_released_bytes_total` | Total number of heap bytes released to OS. | | `go_memstats_heap_sys_bytes` | Number of heap bytes obtained from system. | | `go_memstats_last_gc_time_seconds` | Number of seconds since 1970 of last garbage collection. | | `go_memstats_lookups_total` | Total number of pointer lookups. | | `go_memstats_mallocs_total` | Total number of mallocs. | | `go_memstats_mcache_inuse_bytes` | Number of bytes in use by mcache structures. | | `go_memstats_mcache_sys_bytes` | Number of bytes used for mcache structures obtained from system. | | `go_memstats_mspan_inuse_bytes` | Number of bytes in use by mspan structures. | | `go_memstats_mspan_sys_bytes` | Number of bytes used for mspan structures obtained from system. | | `go_memstats_next_gc_bytes` | Number of heap bytes when next garbage collection will take place. | | `go_memstats_other_sys_bytes` | Number of bytes used for other system allocations. | | `go_memstats_stack_inuse_bytes` | Number of bytes in use by the stack allocator. | | `go_memstats_stack_sys_bytes` | Number of bytes obtained from system for stack allocator. | | `go_memstats_sys_bytes` | Number of bytes obtained by system. Sum of all system allocations. | | `process_cpu_seconds_total` | Total user and system CPU time spent in seconds. | | `process_max_fds` | Maximum number of open file descriptors. | | `process_open_fds` | Number of open file descriptors. | | `process_resident_memory_bytes` | Resident memory size in bytes. | | `process_start_time_seconds` | Start time of the process since unix epoch in seconds. | | `process_virtual_memory_bytes` | Virtual memory size in bytes. | | `promhttp_metric_handler_requests_in_flight` | Current number of scrapes being served. | | `promhttp_metric_handler_requests_total` | Total number of scrapes by HTTP status code. | ### Calico Cloud node (cnx-node) ## [📄️ Configuring cnx-node](https://docs.tigera.io/calico-cloud/reference/component-resources/node/configuration) [Customize cnx-node using environment variables.](https://docs.tigera.io/calico-cloud/reference/component-resources/node/configuration) ## [🗃️ Felix](https://docs.tigera.io/calico-cloud/reference/component-resources/node/felix/) [2 items](https://docs.tigera.io/calico-cloud/reference/component-resources/node/felix/) ### Configuring cnx-node The `cnx-node` container is deployed to every node (on Kubernetes, by a DaemonSet), and runs three internal daemons: - Felix, the Calico daemon that runs on every node and provides endpoints. - BIRD, the BGP daemon that distributes routing information to other nodes. - confd, a daemon that watches the Calico datastore for config changes and updates BIRD’s config files. For manifest-based installations, `cnx-node` is primarily configured through environment variables, typically set in the deployment manifest. Individual nodes may also be updated through the Node custom resource. `cnx-node` can also be configured through the Calico Operator. The rest of this page lists the available configuration options, and is followed by specific considerations for various settings. **Tab: Operator** `cnx-node` does not need to be configured directly when installed by the operator. For a complete operator configuration reference, see [the installation API reference documentation](https://docs.tigera.io/calico-cloud/reference/installation/api). **Tab: Manifest** ## Environment variables[​](#environment-variables) ### Configuring the default IP pool(s)[​](#configuring-the-default-ip-pools) Calico uses IP pools to configure how addresses are allocated to pods, and how networking works for certain sets of addresses. You can see the full schema for IP pools here. `cnx-node` can be configured to create a default IP pool for you, but only if none already exist in the cluster. The following options control the parameters on the created pool. | Environment | Description | Schema | | -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | | CALICO\_IPV4POOL\_CIDR | The IPv4 Pool to create if none exists at start up. It is invalid to define this variable and NO\_DEFAULT\_POOLS. \[Default: First not used in locally of (192.168.0.0/16, 172.16.0.0/16, .., 172.31.0.0/16) ] | IPv4 CIDR | | CALICO\_IPV4POOL\_BLOCK\_SIZE | Block size to use for the IPv4 Pool created at startup. Block size for IPv4 should be in the range 20-32 (inclusive) \[Default: `26`] | int | | CALICO\_IPV4POOL\_IPIP | IPIP Mode to use for the IPv4 Pool created at start up. If set to a value other than `Never`, `CALICO_IPV4POOL_VXLAN` should not be set. \[Default: `Always`] | Always, CrossSubnet, Never ("Off" is also accepted as a synonym for "Never") | | CALICO\_IPV4POOL\_VXLAN | VXLAN Mode to use for the IPv4 Pool created at start up. If set to a value other than `Never`, `CALICO_IPV4POOL_IPIP` should not be set. \[Default: `Never`] | Always, CrossSubnet, Never | | CALICO\_IPV4POOL\_NAT\_OUTGOING | Controls NAT Outgoing for the IPv4 Pool created at start up. \[Default: `true`] | boolean | | CALICO\_IPV4POOL\_NODE\_SELECTOR | Controls the NodeSelector for the IPv4 Pool created at start up. \[Default: `all()`] | [selector](https://docs.tigera.io/calico-cloud/reference/resources/ippool#node-selector) | | CALICO\_IPV6POOL\_CIDR | The IPv6 Pool to create if none exists at start up. It is invalid to define this variable and NO\_DEFAULT\_POOLS. \[Default: ``] | IPv6 CIDR | | CALICO\_IPV6POOL\_BLOCK\_SIZE | Block size to use for the IPv6 POOL created at startup. Block size for IPv6 should be in the range 116-128 (inclusive) \[Default: `122`] | int | | CALICO\_IPV6POOL\_VXLAN | VXLAN Mode to use for the IPv6 Pool created at start up. \[Default: `Never`] | Always, CrossSubnet, Never | | CALICO\_IPV6POOL\_NAT\_OUTGOING | Controls NAT Outgoing for the IPv6 Pool created at start up. \[Default: `false`] | boolean | | CALICO\_IPV6POOL\_NODE\_SELECTOR | Controls the NodeSelector for the IPv6 Pool created at start up. \[Default: `all()`] | [selector](https://docs.tigera.io/calico-cloud/reference/resources/ippool#node-selector) | | CALICO\_IPV4POOL\_DISABLE\_BGP\_EXPORT | Disable exporting routes over BGP for the IPv4 Pool created at start up. \[Default: `false`] | boolean | | CALICO\_IPV6POOL\_DISABLE\_BGP\_EXPORT | Disable exporting routes over BGP for the IPv6 Pool created at start up. \[Default: `false`] | boolean | | NO\_DEFAULT\_POOLS | Prevents Calico Cloud from creating a default pool if one does not exist. \[Default: `false`] | boolean | ### Configuring BGP Networking[​](#configuring-bgp-networking) BGP configuration for Calico nodes is normally configured through the [Node](https://docs.tigera.io/calico-cloud/reference/resources/node), [BGPConfiguration](https://docs.tigera.io/calico-cloud/reference/resources/bgpconfig), and [BGPPeer](https://docs.tigera.io/calico-cloud/reference/resources/bgppeer) resources. `cnx-node` also exposes some options to allow setting certain fields on these objects, as described below. | Environment | Description | Schema | | -------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | | NODENAME | A unique identifier for this host. See [node name determination](#node-name-determination) for more details. | lowercase string | | IP | The IPv4 address to assign this host or detection behavior at startup. Refer to [IP setting](#ip-setting) for the details of the behavior possible with this field. | IPv4 | | IP6 | The IPv6 address to assign this host or detection behavior at startup. Refer to [IP setting](#ip-setting) for the details of the behavior possible with this field. | IPv6 | | IP\_AUTODETECTION\_METHOD | The method to use to autodetect the IPv4 address for this host. This is only used when the IPv4 address is being autodetected. See [IP Autodetection methods](#ip-autodetection-methods) for details of the valid methods. \[Default: `first-found`] | string | | IP6\_AUTODETECTION\_METHOD | The method to use to autodetect the IPv6 address for this host. This is only used when the IPv6 address is being autodetected. See [IP Autodetection methods](#ip-autodetection-methods) for details of the valid methods. \[Default: `first-found`] | string | | AS | The AS number for this node. When specified, the value is saved in the node resource configuration for this host, overriding any previously configured value. When omitted, if an AS number has been previously configured in the node resource, that AS number is used for the peering. When omitted, if an AS number has not yet been configured in the node resource, the node will use the global value (see [example modifying Global BGP settings](https://docs.tigera.io/calico-cloud/networking/configuring/bgp) for details.) | int | | CALICO\_ROUTER\_ID | Sets the `router id` to use for BGP if no IPv4 address is set on the node. For an IPv6-only system, this may be set to `hash`. It then uses the hash of the nodename to create a 4 byte router id. See note below. \[Default: \`\`] | string | | CALICO\_K8S\_NODE\_REF | The name of the corresponding node object in the Kubernetes API. When set, used for correlating this node with events from the Kubernetes API. | string | ### Configuring Datastore Access[​](#configuring-datastore-access) | Environment | Description | Schema | | --------------- | ------------------------------------------- | ------------------ | | DATASTORE\_TYPE | Type of datastore. \[Default: `kubernetes`] | kubernetes, etcdv3 | #### Configuring Kubernetes Datastore Access[​](#configuring-kubernetes-datastore-access) | Environment | Description | Schema | | ------------------ | ------------------------------------------------------------------------------ | ------ | | KUBECONFIG | When using the Kubernetes datastore, the location of a kubeconfig file to use. | string | | K8S\_API\_ENDPOINT | Location of the Kubernetes API. Not required if using kubeconfig. | string | | K8S\_CERT\_FILE | Location of a client certificate for accessing the Kubernetes API. | string | | K8S\_KEY\_FILE | Location of a client key for accessing the Kubernetes API. | string | | K8S\_CA\_FILE | Location of a CA for accessing the Kubernetes API. | string | > **SECONDARY:** When Calico Cloud is configured to use the Kubernetes API as the datastore, the environments used for BGP configuration are ignored—this includes selection of the node AS number (AS) and all of the IP selection options (IP, IP6, IP\_AUTODETECTION\_METHOD, IP6\_AUTODETECTION\_METHOD). ### Configuring Logging[​](#configuring-logging) | Environment | Description | Schema | | ------------------------------ | ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------- | | CALICO\_DISABLE\_FILE\_LOGGING | Disables logging to file. \[Default: "false"] | string | | CALICO\_STARTUP\_LOGLEVEL | The log severity above which startup `cnx-node` logs are sent to the stdout. \[Default: `ERROR`] | DEBUG, INFO, WARNING, ERROR, CRITICAL, or NONE (case-insensitive) | ### Configuring CNI Plugin[​](#configuring-cni-plugin) `cnx-node` has a few options that are configurable based on the CNI plugin and CNI plugin configuration used on the cluster. | Environment | Description | Schema | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | | USE\_POD\_CIDR | Use the Kubernetes `Node.Spec.PodCIDR` field when using host-local IPAM. Requires Kubernetes API datastore. This field is required when using the Kubernetes API datastore with host-local IPAM. \[Default: false] | boolean | | CALICO\_MANAGE\_CNI | Tells Calico to update the kubeconfig file at /host/etc/cni/net.d/calico-kubeconfig on credentials change. \[Default: true] | boolean | ### Other Environment Variables[​](#other-environment-variables) | Environment | Description | Schema | | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | | DISABLE\_NODE\_IP\_CHECK | Skips checks for duplicate Node IPs. This can reduce the load on the cluster when a large number of Nodes are restarting. \[Default: `false`] | boolean | | WAIT\_FOR\_DATASTORE | Wait for connection to datastore before starting. If a successful connection is not made, node will shutdown. \[Default: `false`] | boolean | | CALICO\_NETWORKING\_BACKEND | The networking backend to use. In `bird` mode, Calico will provide BGP networking using the BIRD BGP daemon; VXLAN networking can also be used. In `vxlan` mode, only VXLAN networking is provided; BIRD and BGP are disabled. If set to `none` (also known as policy-only mode), both BIRD and VXLAN are disabled. \[Default: `bird`] | bird, vxlan, none | | CLUSTER\_TYPE | Contains comma delimited list of indicators about this cluster. e.g. k8s, mesos, kubeadm, canal, bgp | string | ## Appendix[​](#appendix) ### Node name determination[​](#node-name-determination) The `cnx-node` must know the name of the node on which it is running. The node name is used to retrieve the [Node resource](https://docs.tigera.io/calico-cloud/reference/resources/node) configured for this node if it exists, or to create a new node resource representing the node if it does not. It is also used to associate the node with per-node [BGP configuration](https://docs.tigera.io/calico-cloud/reference/resources/bgpconfig), [felix configuration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig), and endpoints. When launched, the `cnx-node` container sets the node name according to the following order of precedence: 1. The value specified in the `NODENAME` environment variable, if set. 2. The value specified in `/var/lib/calico/nodename`, if it exists. 3. The value specified in the `HOSTNAME` environment variable, if set. 4. The hostname as returned by the operating system, converted to lowercase. Once the node has determined its name, the value will be cached in `/var/lib/calico/nodename` for future use. For example, if given the following conditions: - `NODENAME=""` - `/var/lib/calico/nodename` does not exist - `HOSTNAME="host-A"` - The operating system returns "host-A.internal.myorg.com" for the hostname cnx-node will use "host-a" for its name and will write the value in `/var/lib/calico/nodename`. If cnx-node is then restarted, it will use the cached value of "host-a" read from the file on disk. ### IP setting[​](#ip-setting) The IP (for IPv4) and IP6 (for IPv6) environment variables are used to set, force autodetection, or disable auto detection of the address for the appropriate IP version for the node. When the environment variable is set, the address is saved in the [node resource configuration](https://docs.tigera.io/calico-cloud/reference/resources/node) for this host, overriding any previously configured value. calico/node will attempt to detect subnet information from the host, and augment the provided address if possible. #### IP setting special case values[​](#ip-setting-special-case-values) There are several special case values that can be set in the IP(6) environment variables, they are: - Not set or empty string: Any previously set address on the node resource will be used. If no previous address is set on the node resource the two versions behave differently: - IP will do autodetection of the IPv4 address and set it on the node resource. - IP6 will not do autodetection. - `autodetect`: Autodetection will always be performed for the IP address and the detected address will overwrite any value configured in the node resource. - `none`: Autodetection will not be performed (this is useful to disable IPv4). ### IP autodetection methods[​](#ip-autodetection-methods) When Calico Cloud is used for routing, each node must be configured with an IPv4 address and/or an IPv6 address that will be used to route between nodes. To eliminate node specific IP address configuration, the `cnx-node` container can be configured to autodetect these IP addresses. In many systems, there might be multiple physical interfaces on a host, or possibly multiple IP addresses configured on a physical interface. In these cases, there are multiple addresses to choose from and so autodetection of the correct address can be tricky. The IP autodetection methods are provided to improve the selection of the correct address, by limiting the selection based on suitable criteria for your deployment. The following sections describe the available IP autodetection methods. #### first-found[​](#first-found) The `first-found` option enumerates all interface IP addresses and returns the first valid IP address (based on IP version and type of address) on the first valid interface. Certain known "local" interfaces are omitted, such as the docker bridge. The order that both the interfaces and the IP addresses are listed is system dependent. This is the default detection method. However, since this method only makes a very simplified guess, it is recommended to either configure the node with a specific IP address, or to use one of the other detection methods. e.g. ```text IP_AUTODETECTION_METHOD=first-found IP6_AUTODETECTION_METHOD=first-found ``` #### kubernetes-internal-ip[​](#kubernetes-internal-ip) The `kubernetes-internal-ip` method will select the first internal IP address listed in the Kubernetes node's `Status.Addresses` field Example: ```text IP_AUTODETECTION_METHOD=kubernetes-internal-ip IP6_AUTODETECTION_METHOD=kubernetes-internal-ip ``` #### can-reach=DESTINATION[​](#can-reachdestination) The `can-reach` method uses your local routing to determine which IP address will be used to reach the supplied destination. Both IP addresses and domain names may be used. Example using IP addresses: ```text IP_AUTODETECTION_METHOD=can-reach=8.8.8.8 IP6_AUTODETECTION_METHOD=can-reach=2001:4860:4860::8888 ``` Example using domain names: ```text IP_AUTODETECTION_METHOD=can-reach=www.google.com IP6_AUTODETECTION_METHOD=can-reach=www.google.com ``` #### interface=INTERFACE-REGEX[​](#interfaceinterface-regex) The `interface` method uses the supplied interface [regular expression](https://pkg.go.dev/regexp) to enumerate matching interfaces and to return the first IP address on the first matching interface. The order that both the interfaces and the IP addresses are listed is system dependent. Example with valid IP address on interface eth0, eth1, eth2 etc.: ```text IP_AUTODETECTION_METHOD=interface=eth.* IP6_AUTODETECTION_METHOD=interface=eth.* ``` #### skip-interface=INTERFACE-REGEX[​](#skip-interfaceinterface-regex) The `skip-interface` method uses the supplied interface [regular expression](https://pkg.go.dev/regexp) to exclude interfaces and to return the first IP address on the first interface that does not match. The order that both the interfaces and the IP addresses are listed is system dependent. Example with valid IP address on interface exclude enp6s0f0, eth0, eth1, eth2 etc.: ```text IP_AUTODETECTION_METHOD=skip-interface=enp6s0f0,eth.* IP6_AUTODETECTION_METHOD=skip-interface=enp6s0f0,eth.* ``` #### cidr=CIDR[​](#cidrcidr) The `cidr` method will select any IP address from the node that falls within the given CIDRs. For example: Example: ```text IP_AUTODETECTION_METHOD=cidr=10.0.1.0/24,10.0.2.0/24 IP6_AUTODETECTION_METHOD=cidr=2001:4860::0/64 ``` ### Node readiness[​](#node-readiness) The `calico/node` container supports an exec readiness endpoint. To access this endpoint, use the following command. ```bash docker exec calico-node /bin/calico-node [flag] ``` Substitute `[flag]` with one or more of the following. - `-bird-ready` - `-bird6-ready` - `-felix-ready` The BIRD readiness endpoint ensures that the BGP mesh is healthy by verifying that all BGP peers are established and no graceful restart is in progress. ### Setting `CALICO_ROUTER_ID` for IPv6 only system[​](#setting-calico_router_id-for-ipv6-only-system) Setting CALICO\_ROUTER\_ID to value `hash` will use a hash of the configured nodename for the router ID. This should only be used in IPv6-only systems with no IPv4 address to use for the router ID. Since each node chooses its own router ID in isolation, it is possible for two nodes to pick the same ID resulting in a clash. The probability of such a clash grows with cluster size so this feature should not be used in a large cluster (500+ nodes). ### Felix ## [📄️ Configuring Felix](https://docs.tigera.io/calico-cloud/reference/component-resources/node/felix/configuration) [Configure Felix, the daemon that runs on every machine that provides endpoints.](https://docs.tigera.io/calico-cloud/reference/component-resources/node/felix/configuration) ## [📄️ Monitoring Felix with Prometheus](https://docs.tigera.io/calico-cloud/reference/component-resources/node/felix/prometheus) [Review metrics for the Felix component if you are using Prometheus.](https://docs.tigera.io/calico-cloud/reference/component-resources/node/felix/prometheus) ### Configuring Felix **Tab: Operator** If you have installed Calico using the operator, you cannot modify the environment provided to felix directly. To configure felix, see the [FelixConfiguration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig) resource instead. **Tab: Manifest** > **SECONDARY:** The following tables detail the configuration file and environment variable parameters. For `FelixConfiguration` resource settings, refer to [Felix Configuration Resource](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig). Configuration for Felix is read from one of four possible locations, in order, as follows. 1. Environment variables. 2. The Felix configuration file. 3. Host-specific `FelixConfiguration` resources (`node.`). 4. The global `FelixConfiguration` resource (`default`). The value of any configuration parameter is the value read from the *first* location containing a value. For example, if an environment variable contains a value, it takes top precedence. If not set in any of these locations, most configuration parameters have defaults, and it should be rare to have to explicitly set them. The full list of parameters which can be set is as follows. ### General configuration[​](#general-configuration) | Configuration file parameter | Environment variable | Description | Schema | | ----------------------------------- | ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------- | | `DataplaneWatchdogTimeout` | `FELIX_DATAPLANEWATCHDOGTIMEOUT` | Deprecated: superseded by `HealthTimeoutOverrides`. Timeout before the main data plane goroutine is determined to have hung and Felix will report non-live and non-ready. Can be increased if the liveness check incorrectly fails (for example if Felix is running slowly on a heavily loaded system). \[Default: `90`] | int | | `AwsSrcDstCheck` | `FELIX_AWSSRCDSTCHECK` | Set the [source-destination-check](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_NAT_Instance.html#EIP_Disable_SrcDestCheck) when using AWS EC2 instances. Check [IAM role and profile configuration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig#aws-iam-rolepolicy-for-source-destination-check-configuration) for setting the necessary permission for this setting to work. \[Default: `DoNothing`] | `DoNothing`, `Disable`, `Enable` | | `AWSSecondaryIPSupport` | `FELIX_AWSSECONDARYIPSUPPORT` | Controls whether Felix will create secondary AWS ENIs for AWS-backed IP pools. This feature is documented in the [egress gateways on AWS guide](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-aws). Should only be enabled on AWS. \[Default: "Disabled"] | `Enabled`, `EnabledENIPerWorkload`, `Disabled` | | `AWSSecondaryIPRoutingRulePriority` | `FELIX_AWSSECONDARYIPROUTINGRULEPRIORITY` | Controls the priority of the policy-based routing rules used to implement AWS-backed IP addresses. Should only be changed to avoid conflicts if your nodes have additional policy based routing rules. \[Default: 101] | int | | `AWSRequestTimeout` | `FELIX_AWSREQUESTTIMEOUT` Timeout used for communicating with the AWS API (seconds). \[Default: "30"] | int | | | `DatastoreType` | `FELIX_DATASTORETYPE` | The datastore that Felix should read endpoints and policy information from. \[Default: `etcdv3`] | `etcdv3`, `kubernetes` | | `DeviceRouteSourceAddress` | `FELIX_DEVICEROUTESOURCEADDRESS` | IPv4 address to use as the source hint on device routes programmed by Felix \[Default: No source hint is set on programmed routes and for local traffic from host to workload the source address will be chosen by the kernel.] | `` | | `DeviceRouteSourceAddressIPv6` | `FELIX_DEVICEROUTESOURCEADDRESSIPV6` | IPv6 address to use as the source hint on device routes programmed by Felix \[Default: No source hint is set on programmed routes and for local traffic from host to workload the source address will be chosen by the kernel.] | `` | | `DeviceRouteProtocol` | `FELIX_DEVICEROUTEPROTOCOL` | This defines the route protocol added to programmed device routes. \[Default: `RTPROT_BOOT`] | int | | `DisableConntrackInvalidCheck` | `FELIX_DISABLECONNTRACKINVALIDCHECK` | Disable the dropping of packets that aren't either a valid handshake or part of an established connection. \[Default: `false`] | boolean | | `EndpointReportingDelaySecs` | `FELIX_ENDPOINTREPORTINGDELAYSECS` | Set the endpoint reporting delay between status check intervals, in seconds. Only used if endpoint reporting is enabled. \[Default: `1`] | int | | `EndpointReportingEnabled` | `FELIX_ENDPOINTREPORTINGENABLED` | Enable the endpoint status reporter. \[Default: `false`] | boolean | | `ExternalNodesCIDRList` | `FELIX_EXTERNALNODESCIDRLIST` | Comma-delimited list of IPv4 or CIDR of external-non-calico-nodes from which IPIP traffic is accepted by calico-nodes. \[Default: ""] | string | | `FailsafeInboundHostPorts` | `FELIX_FAILSAFEINBOUNDHOSTPORTS` | List of PortProto struct objects including UDP/TCP/SCTP ports and CIDRs that Felix will allow incoming traffic to host endpoints on irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. For backwards compatibility, if the protocol is not specified, it defaults to `"tcp"`. If a CIDR is not specified, it will allow traffic from all addresses. To disable all inbound host ports, use the value `[]`. The default value allows ssh access, DHCP, BGP, etcd and the Kubernetes API. \[Default: `[{"port":22,"protocol":"tcp"},{"port":68,"protocol":"udp"},{"port":179,"protocol":"tcp"},{"port":2379,"protocol":"tcp"}, {"port":2380,"protocol":"tcp"}, {"port":5473,"protocol":"tcp"}, {"port":6443,"protocol":"tcp"}, {"port":6666,"protocol":"tcp"}, {"port":6667,"protocol":"tcp"}]`] | list | | `FailsafeOutboundHostPorts` | `FELIX_FAILSAFEOUTBOUNDHOSTPORTS` | List of PortProto struct objects including UDP/TCP/SCTP ports and CIDRs that Felix will allow outgoing traffic from host endpoints to irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. For backwards compatibility, if the protocol is not specified, it defaults to `"tcp"`. If a CIDR is not specified, it will allow traffic from all addresses. To disable all outbound host ports, use the value `[]`. The default value opens etcd's standard ports to ensure that Felix does not get cut off from etcd as well as allowing DHCP, DNS, BGP and the Kubernetes API. \[Default: `[{"port":53,"protocol":"udp"},{"port":67,"protocol":"udp"}, {"port":179,"protocol":"tcp"}, {"port":2379,"protocol":"tcp"}, {"port":2380,"protocol":"tcp"}, {"port":5473,"protocol":"tcp"}, {"port":6443,"protocol":"tcp"}, {"port": 6666,"protocol":"tcp"}, {"port":6667,"protocol":"tcp"}]`] | list | | `FelixHostname` | `FELIX_FELIXHOSTNAME` | The hostname Felix reports to the plugin. Should be used if the hostname Felix autodetects is incorrect or does not match what the plugin will expect. \[Default: `socket.gethostname()`] | string | | `HealthEnabled` | `FELIX_HEALTHENABLED` | When enabled, exposes felix health information via an http endpoint. | boolean | | `HealthHost` | `FELIX_HEALTHHOST` | The address on which Felix will respond to health requests. \[Default: `localhost`] | string | | `HealthPort` | `FELIX_HEALTHPORT` | The port on which Felix will respond to health requests. \[Default: `9099`] | int | | `HealthTimeoutOverrides` | `FELIX_HEALTHTIMEOUTOVERRIDES` | Allows the internal watchdog timeouts of individual subcomponents to be overridden; example: "InternalDataplaneMainLoop=30s,CalculationGraph=2m". This is useful for working around "false positive" liveness timeouts that can occur in particularly stressful workloads or if CPU is constrained. For a list of active subcomponents, see Felix's logs. \[Default: \`\`] | Comma-delimited list of key/value pairs where the values are durations: `1s`, `10s`, `5m`, etc. | | `IpInIpEnabled` | `FELIX_IPINIPENABLED` | Optional, you shouldn't need to change this setting as Felix calculates if IPIP should be enabled based on the existing IP Pools. When set, this overrides whether Felix should configure an IPinIP interface on the host. When explicitly disabled in FelixConfiguration, Felix will not clean up addresses from the `tunl0` interface (use this if you need to add addresses to that interface and don't want to have them removed). \[Default: unset] | optional boolean | | `IpInIpMtu` | `FELIX_IPINIPMTU` | The MTU to set on the IPIP tunnel device. Zero value means auto-detect. See [Configuring MTU](https://docs.tigera.io/calico-cloud/networking/configuring/mtu) \[Default: `0`] | int | | `IPv4VXLANTunnelAddr` | | IP address of the IPv4 VXLAN tunnel. This is system configured and should not be updated manually. | string | | `LogFilePath` | `FELIX_LOGFILEPATH` | The full path to the Felix log. Set to `none` to disable file logging. \[Default: `/var/log/calico/felix.log`] | string | | `LogSeverityFile` | `FELIX_LOGSEVERITYFILE` | The log severity above which logs are sent to the log file. \[Default: `Info`] | `Debug`, `Info`, `Warning`, `Error`, `Fatal` | | `LogSeverityScreen` | `FELIX_LOGSEVERITYSCREEN` | The log severity above which logs are sent to the stdout. \[Default: `Info`] | `Debug`, `Info`, `Warning`, `Error`, `Fatal` | | `LogSeveritySys` | `FELIX_LOGSEVERITYSYS` | The log severity above which logs are sent to the syslog. Set to `none` for no logging to syslog. \[Default: `Info`] | `Debug`, `Info`, `Warning`, `Error`, `Fatal` | | `LogDebugFilenameRegex` | `FELIX_LOGDEBUGFILENAMEREGEX` | Controls which source code files have their Debug log output included in the logs. Only logs from files with names that match the given regular expression are included. The filter only applies to Debug level logs. \[Default: `""`] | regex | | `PolicySyncPathPrefix` | `FELIX_POLICYSYNCPATHPREFIX` | File system path where Felix notifies services of policy changes over Unix domain sockets. This is required only if you're configuring [L7 logs](https://docs.tigera.io/calico-cloud/observability/elastic/l7/configure), or [egress gateways](https://docs.tigera.io/calico-cloud/networking/egress/). Set to `""` to disable. \[Default: `""`] | string | | `PrometheusGoMetricsEnabled` | `FELIX_PROMETHEUSGOMETRICSENABLED` | Set to `false` to disable Go runtime metrics collection, which the Prometheus client does by default. This reduces the number of metrics reported, reducing Prometheus load. \[Default: `true`] | boolean | | `PrometheusMetricsEnabled` | `FELIX_PROMETHEUSMETRICSENABLED` | Set to `true` to enable the Prometheus metrics server in Felix. \[Default: `false`] | boolean | | `PrometheusMetricsHost` | `FELIX_PROMETHEUSMETRICSHOST` | TCP network address that the Prometheus metrics server should bind to. \[Default: `""`] | string | | `PrometheusMetricsPort` | `FELIX_PROMETHEUSMETRICSPORT` | TCP port that the Prometheus metrics server should bind to. \[Default: `9091`] | int | | `PrometheusProcessMetricsEnabled` | `FELIX_PROMETHEUSPROCESSMETRICSENABLED` | Set to `false` to disable process metrics collection, which the Prometheus client does by default. This reduces the number of metrics reported, reducing Prometheus load. \[Default: `true`] | boolean | | `PrometheusWireguardMetricsEnabled` | `FELIX_PROMETHEUSWIREGUARDMETRICSENABLED` | Set to `false` to disable wireguard device metrics collection, which Felix does by default. \[Default: `true`] | boolean | | `RemoveExternalRoutes` | `FELIX_REMOVEEXTERNALROUTES` | Whether or not to remove device routes that have not been programmed by Felix. Disabling this will allow external applications to also add device routes. \[Default: `true`] | bool | | `ReportingIntervalSecs` | `FELIX_REPORTINGINTERVALSECS` | Interval at which Felix reports its status into the datastore. 0 means disabled and is correct for Kubernetes-only clusters. Must be non-zero in OpenStack deployments. \[Default: `30`] | int | | `ReportingTTLSecs` | `FELIX_REPORTINGTTLSECS` | Time-to-live setting for process-wide status reports. \[Default: `90`] | int | | `RouteTableRange` | `FELIX_ROUTETABLERANGE` | *deprecated in favor of `RouteTableRanges`* Calico programs additional Linux route tables for various purposes. `RouteTableRange` specifies the indices of the route tables that Calico should use. \[Default: `""`] | `-` | | `RouteTableRanges` | `FELIX_ROUTETABLERANGES` | Calico programs additional Linux route tables for various purposes. `RouteTableRanges` specifies a set of table index ranges that Calico should use. Deprecates `RouteTableRange`, overrides `RouteTableRange`. \[Default: `"1-250"`] | `-,-,...` | | `RouteSyncDisabled` | `FELIX_ROUTESYNCDISABLED` | Set to `true` to disable Calico programming routes to local workloads. \[Default: `false`] | boolean | | `VXLANEnabled` | `FELIX_VXLANENABLED` | Optional, you shouldn't need to change this setting as Felix calculates if VXLAN should be enabled based on the existing IP Pools. When set, this overrides whether Felix should create the VXLAN tunnel device for VXLAN networking. \[Default: unset] | optional boolean | | `VXLANMTU` | `FELIX_VXLANMTU` | The MTU to set on the IPv4 VXLAN tunnel device. Zero value means auto-detect. Also controls NodePort MTU when eBPF enabled. See [Configuring MTU](https://docs.tigera.io/calico-cloud/networking/configuring/mtu) \[Default: `0`] | int | | `VXLANMTUV6` | `FELIX_VXLANMTUV6` | The MTU to set on the IPv6 VXLAN tunnel device. Zero value means auto-detect. Also controls NodePort MTU when eBPF enabled. See [Configuring MTU](https://docs.tigera.io/calico-cloud/networking/configuring/mtu) \[Default: `0`] | int | | `VXLANPort` | `FELIX_VXLANPORT` | The UDP port to use for VXLAN. \[Default: `4789`] | int | | `VXLANTunnelMACAddr` | | MAC address of the IPv4 VXLAN tunnel. This is system configured and should not be updated manually. | string | | `VXLANVNI` | `FELIX_VXLANVNI` | The virtual network ID to use for VXLAN. \[Default: `4096`] | int | | `AllowVXLANPacketsFromWorkloads` | `FELIX_ALLOWVXLANPACKETSFROMWORKLOADS` | Set to `true` to allow VXLAN encapsulated traffic from workloads. \[Default: `false`] | boolean | | `AllowIPIPPacketsFromWorkloads` | `FELIX_ALLOWIPIPPACKETSFROMWORKLOADS` | Set to `true` to allow IPIP encapsulated traffic from workloads. \[Default: `false`] | boolean | | `TyphaAddr` | `FELIX_TYPHAADDR` | IPv4 address at which Felix should connect to Typha. \[Default: none] | string | | `TyphaK8sServiceName` | `FELIX_TYPHAK8SSERVICENAME` | Name of the Typha Kubernetes service | string | | `Ipv6Support` | `FELIX_IPV6SUPPORT` | Enable Calico Cloud networking and security for IPv6 traffic as well as for IPv4. | boolean | | `RouteSource` | `FELIX_ROUTESOURCE` | Where Felix gets is routing information from for VXLAN and the BPF data plane. The CalicoIPAM setting is more efficient because it supports route aggregation, but it only works when Calico's IPAM or host-local IPAM is in use. Use the WorkloadIPs setting if you are using Calico's VXLAN or BPF data plane and not using Calico IPAM or host-local IPAM. \[Default: "CalicoIPAM"] | 'CalicoIPAM', or 'WorkloadIPs' | | `mtuIfacePattern` | `FELIX_MTUIFACEPATTERN` | Pattern used to discover the host's interface for MTU auto-detection. \[Default: `^((en\|wl\|ww\|sl\|ib)[Pcopsvx].*\|(eth\|wlan\|wwan).*)`] | regex | | `TPROXYMode` | `FELIX_TPROXYMODE` | Sets transparent proxying mode. \[Default: "Disabled"] | 'Disabled', 'Enabled' | | `TPROXYPort` | `FELIX_TPROXYPORT` | What local ports is the proxied traffic sent to. \[Default: `16001`] | int | | `FeatureDetectOverride` | `FELIX_FEATUREDETECTOVERRIDE` | Is used to override the feature detection. Values are specified in a comma separated list with no spaces, example; "SNATFullyRandom=true,MASQFullyRandom=false,RestoreSupportsLock=true,IPIPDeviceIsL3=true. "true" or "false" will force the feature, empty or omitted values are auto-detected. \[Default: `""`] | string | ### etcd datastore configuration[​](#etcd-datastore-configuration) | Configuration parameter | Environment variable | Description | Schema | | ----------------------- | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | | `EtcdCaFile` | `FELIX_ETCDCAFILE` | Path to the file containing the root certificate of the certificate authority (CA) that issued the etcd server certificate. Configures Felix to trust the CA that signed the root certificate. The file may contain multiple root certificates, causing Felix to trust each of the CAs included. To disable authentication of the server by Felix, set the value to `none`. \[Default: `/etc/ssl/certs/ca-certificates.crt`] | string | | `EtcdCertFile` | `FELIX_ETCDCERTFILE` | Path to the file containing the client certificate issued to Felix. Enables Felix to participate in mutual TLS authentication and identify itself to the etcd server. Example: `/etc/felix/cert.pem` (optional) | string | | `EtcdEndpoints` | `FELIX_ETCDENDPOINTS` | Comma-delimited list of etcd endpoints to connect to. Example: `http://127.0.0.1:2379,http://127.0.0.2:2379`. | `://:` | | `EtcdKeyFile` | `FELIX_ETCDKEYFILE` | Path to the file containing the private key matching Felix's client certificate. Enables Felix to participate in mutual TLS authentication and identify itself to the etcd server. Example: `/etc/felix/key.pem` (optional) | string | ### Kubernetes API datastore configuration[​](#kubernetes-api-datastore-configuration) The Kubernetes API datastore driver reads its configuration from Kubernetes-provided environment variables. ### iptables data plane configuration[​](#iptables-data-plane-configuration) | Configuration parameter | Environment variable | Description | Schema | | ------------------------------------ | ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | | `ChainInsertMode` | `FELIX_CHAININSERTMODE` | Controls whether Felix hooks the kernel's top-level iptables chains by inserting a rule at the top of the chain or by appending a rule at the bottom. `Insert` is the safe default since it prevents Calico Cloud's rules from being bypassed. If you switch to `Append` mode, be sure that the other rules in the chains signal acceptance by falling through to the Calico Cloud rules, otherwise the Calico Cloud policy will be bypassed. \[Default: `Insert`] | `Insert`, `Append` | | `DefaultEndpointToHostAction` | `FELIX_DEFAULTENDPOINTTOHOSTACTION` | This parameter controls what happens to traffic that goes from a workload endpoint to the host itself (after the traffic hits the endpoint egress policy). By default Calico Cloud blocks traffic from workload endpoints to the host itself with an iptables `Drop` action. If you want to allow some or all traffic from endpoint to host, set this parameter to `Return` or `Accept`. Use `Return` if you have your own rules in the iptables "INPUT" chain; Calico Cloud will insert its rules at the top of that chain, then `Return` packets to the "INPUT" chain once it has completed processing workload endpoint egress policy. Use `Accept` to unconditionally accept packets from workloads after processing workload endpoint egress policy. \[Default: `Drop`] | `Drop`, `Return`, `Accept` | | `GenericXDPEnabled` | `FELIX_GENERICXDPENABLED` | When enabled, Felix can fallback to the non-optimized `generic` XDP mode. This should only be used for testing since it doesn't improve performance over the non-XDP mode. \[Default: `false`] | boolean | | `InterfaceExclude` | `FELIX_INTERFACEEXCLUDE` | A comma-separated list of interface names that should be excluded when Felix is resolving host endpoints. The default value ensures that Felix ignores Kubernetes' internal `kube-ipvs0` device. If you want to exclude multiple interface names using a single value, the list supports regular expressions. For regular expressions you must wrap the value with `/`. For example having values `/^kube/,veth1` will exclude all interfaces that begin with `kube` and also the interface `veth1`. \[Default: `kube-ipvs0`] | string | | `IpsetsRefreshInterval` | `FELIX_IPSETSREFRESHINTERVAL` | Period, in seconds, at which Felix re-checks the IP sets in the data plane to ensure that no other process has accidentally broken Calico Cloud's rules. Set to 0 to disable IP sets refresh. \[Default: `10`] | int | | `IptablesBackend` | `FELIX_IPTABLESBACKEND` | This parameter controls which variant of iptables Felix uses. Set this to `Auto` for auto detection of the backend. If a specific backend is needed then use `nft` for hosts using a netfilter backend or `Legacy` for others. \[Default: `Auto`] | `Legacy`, `NFT`, `Auto` | | `IptablesFilterAllowAction` | `FELIX_IPTABLESFILTERALLOWACTION` | This parameter controls what happens to traffic that is allowed by a Felix policy chain in the iptables filter table (i.e., a normal policy chain). The default will immediately `Accept` the traffic. Use `Return` to send the traffic back up to the system chains for further processing. \[Default: `Accept`] | `Accept`, `Return` | | `IptablesLockFilePath` | `FELIX_IPTABLESLOCKFILEPATH` | *Deprecated:* For iptables versions prior to v1.6.2, location of the iptables lock file (later versions of iptables always use value "/run/xtables.lock"). You may need to change this if the lock file is not in its standard location (for example if you have mapped it into Felix's container at a different path). \[Default: `/run/xtables.lock`] | string | | `IptablesLockProbeIntervalMillis` | `FELIX_IPTABLESLOCKPROBEINTERVALMILLIS` | Time, in milliseconds, that Felix will wait between attempts to acquire the iptables lock if it is not available. Lower values make Felix more responsive when the lock is contended, but use more CPU. \[Default: `50`] | int | | `IptablesLockTimeoutSecs` | `FELIX_IPTABLESLOCKTIMEOUTSECS` | Time, in seconds, that Felix will wait for the iptables lock. Versions of iptables prior to v1.6.2 support disabling the iptables lock by setting this value to 0; v1.6.2 and above do not so Felix will default to 10s if a non-positive number is used. To use this feature, Felix must share the iptables lock file with all other processes that also take the lock. When running Felix inside a container, this typically requires the file /run/xtables.lock on the host to be mounted into the `cnx-node` or `calico/felix` container. \[Default: `0` disabled for iptables \` | | `NetlinkTimeoutSecs` | `FELIX_NETLINKTIMEOUTSECS` | Time, in seconds, that Felix will wait for netlink (i.e. routing table list/update) operations to complete before giving up and retrying. \[Default: `10`] | float | | `RouteRefreshInterval` | `FELIX_ROUTEREFRESHINTERVAL` | Period, in seconds, at which Felix re-checks the routes in the data plane to ensure that no other process has accidentally broken Calico Cloud's rules. Set to 0 to disable route refresh. \[Default: `90`] | int | | `ServiceLoopPrevention` | `FELIX_SERVICELOOPPREVENTION` | When [service IP advertisement is enabled](https://docs.tigera.io/calico-cloud/networking/configuring/advertise-service-ips), prevent routing loops to service IPs that are not in use, by dropping or rejecting packets that do not get DNAT'd by kube-proxy. Unless set to "Disabled", in which case such routing loops continue to be allowed. \[Default: `Drop`] | `Drop`, `Reject`, `Disabled` | | `WorkloadSourceSpoofing` | `FELIX_WORKLOADSOURCESPOOFING` | Controls whether pods can enable source IP address spoofing with the `cni.projectcalico.org/allowedSourcePrefixes` annotation. When set to `Any`, pods can use this annotation to send packets from any IP address. \[Default: `Disabled`] | `Any`, `Disabled` | | `XDPRefreshInterval` | `FELIX_XDPREFRESHINTERVAL` | Period, in seconds, at which Felix re-checks the XDP state in the data plane to ensure that no other process has accidentally broken Calico Cloud's rules. Set to 0 to disable XDP refresh. \[Default: `90`] | int | | `XDPEnabled` | `FELIX_XDPENABLED` | Enable XDP acceleration for host endpoint policies. \[Default: `true`] | boolean | ### eBPF data plane configuration[​](#ebpf-data-plane-configuration) eBPF data plane mode uses the Linux Kernel's eBPF virtual machine to implement networking and policy instead of iptables. When BPFEnabled is set to `true`, Felix will: - Require a v5.3 Linux kernel. - Implement policy with eBPF programs instead of iptables. - Activate its embedded implementation of `kube-proxy` to implement Kubernetes service load balancing. - Disable support for IPv6. See [Enable the eBPF data plane](https://docs.tigera.io/calico-cloud/operations/ebpf/enabling-ebpf) for step-by step instructions to enable this feature. \| Configuration parameter / Environment variable | Description | Schema | Default | | ----------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | -------------- | ------ | ------ | ----------------- | | BPFEnabled /\ FELIX\_BPFENABLED | Enable eBPF data plane mode. eBPF mode has a number of limitations, see the [HOWTO guide](https://docs.tigera.io/calico-cloud/operations/ebpf/enabling-ebpf). | true, false | false | | BPFDisableUnprivileged /\ FELIX\_BPFDISABLEUNPRIVILEGED | If true, Felix sets the kernel.unprivileged\_bpf\_disabled sysctl to disable unprivileged use of BPF. This ensures that unprivileged users cannot access Calico's BPF maps and cannot insert their own BPF programs to interfere with the ones that Calico Cloud installs. | true, false | true | | BPFLogLevel /\ FELIX\_BPFLOGLEVEL | The log level used by the BPF programs. The logs are emitted to the BPF trace pipe, accessible with the command `tc exec BPF debug`. | Off,Info,Debug | Off | | BPFDataIfacePattern /\ FELIX\_BPFDATAIFACEPATTERN | Controls which interfaces Felix should attach BPF programs to catch traffic to/from the external network. This needs to match the interfaces that Calico workload traffic flows over as well as any interfaces that handle incoming traffic to NodePorts and services from outside the cluster. It should not match the workload interfaces (usually named cali...).. | regular expression | `^(en[opsvx].\* | eth.\* | tunl0$ | wireguard.cali$)` | | BPFConnectTimeLoadBalancingEnabled /\ FELIX\_BPFCONNECTTIMELOADBALANCINGENABLED | Controls whether Felix installs the connect-time load balancer. In the current release, the connect-time load balancer is required for the host to reach kubernetes services. | true,false | true | | BPFExternalServiceMode /\ FELIX\_BPFEXTERNALSERVICEMODE | Controls how traffic from outside the cluster to NodePorts and ClusterIPs is handled. In Tunnel mode, packet is tunneled from the ingress host to the host with the backing pod and back again. In DSR mode, traffic is tunneled to the host with the backing pod and then returned directly; this requires a network that allows direct return. | Tunnel,DSR | Tunnel | | BPFExtToServiceConnmark /\ FELIX\_BPFEXTTOSERVICECONNMARK | Controls a 32bit mark that is set on connections from an external client to a local service. This mark allows us to control how packets of that connection are routed within the host and how is routing interpreted by RPF check. | int | 0 | | BPFKubeProxyIptablesCleanupEnabled /\ FELIX\_BPFKUBEPROXYIPTABLESCLEANUPENABLED | Controls whether Felix will clean up the iptables rules created by the Kubernetes `kube-proxy`; should only be enabled if `kube-proxy` is not running. | true,false | true | | BPFKubeProxyMinSyncPeriod /\ FELIX\_BPFKUBEPROXYMINSYNCPERIOD | Controls the minimum time between data plane updates for Felix's embedded `kube-proxy` implementation. | seconds | `1` | | BPFKubeProxyEndpointSlicesEnabled /\ FELIX\_BPFKUBEPROXYENDPOINTSLICESENABLED | Controls whether Felix's embedded kube-proxy derives its services from Kubernetes' EndpointSlices resources. Using EndpointSlices is more efficient but it requires EndpointSlices support to be enabled at the Kubernetes API server. | true,false | false | | BPFMapSizeConntrack /\ FELIX\_BPFMapSizeConntrack | Controls the size of the conntrack map. This map must be large enough to hold an entry for each active connection. Warning: changing the size of the conntrack map can cause disruption. | int | 512000 | | BPFMapSizeNATFrontend /\ FELIX\_BPFMapSizeNATFrontend | Controls the size of the NAT frontend map. FrontendMap should be large enough to hold an entry for each nodeport, external IP and each port in each service. | int | 65536 | | BPFMapSizeNATBackend /\ FELIX\_BPFMapSizeNATBackend | Controls the size of the NAT backend map. This is the total number of endpoints. This is mostly more than the size of the number of services. | int | 262144 | | BPFMapSizeNATAffinity /\ FELIX\_BPFMapSizeNATAffinity | Controls the size of the NAT affinity map. | int | 65536 | | BPFMapSizeIPSets /\ FELIX\_BPFMapSizeIPSets | Controls the size of the IPSets map. The IP sets map must be large enough to hold an entry for each endpoint matched by every selector in the source/destination matches in network policy. Selectors such as "all()" can result in large numbers of entries (one entry per endpoint in that case). | int | 1048576 | | BPFMapSizeRoute /\ FELIX\_BPFMapSizeRoute | Controls the size of the route map. The routes map should be large enough to hold one entry per workload and a handful of entries per host (enough to cover its own IPs and tunnel IPs). | int | 262144 | | BPFHostConntrackBypass /\ FELIX\_BPFHostConntrackBypass | Controls whether to bypass Linux conntrack in BPF mode for workloads and services. | true,false | true | | BPFPolicyDebugEnabled /\ FELIX\_BPFPOLICYDEBUGENABLED | In eBPF data plane mode, Felix records detailed information about the BPF policy programs, which can be examined with the calico-bpf command-line tool. | true, false | true | ### Windows-specific configuration[​](#windows-specific-configuration) | Configuration parameter | Environment variable | Description | Schema | Default | | ------------------------------- | --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------------------------------------------- | | windowsFlowLogsFileDirectory | FELIX\_WINDOWSFLOWLOGSFILEDIRECTORY | Set the directory where flow logs files are stored on Windows nodes. This parameter only takes effect when `flowLogsFileEnabled` is set to `true`. | string | `c:\\TigeraCalico\\flowlogs` | | windowsFlowLogsPositionFilePath | FELIX\_WINDOWSFLOWLOGSPOSITIONFILEPATH | Specify the position of the external pipeline that reads flow logs on Windows nodes. This parameter only takes effect when `FlowLogsDynamicAggregationEnabled` is set to `true`. | string | `c:\\TigeraCalico\\flowlogs\\flows.log.pos` | | windowsStatsDumpFilePath | FELIX\_WINDOWSTATSDUMPFILEPATH | Specify the position of the file used for dumping flow log statistics on Windows nodes. Note this is an internal setting that users shouldn't need to modify. | string | `c:\\TigeraCalico\\stats\\dump` | | WindowsDNSCacheFile | FELIX\_WINDOWSDNSCACHEFILE | Specify the name of the file that Felix uses to preserve learned DNS information when restarting. | string | `c:\\TigeraCalico\\felix-dns-cache.txt` | | WindowsDNSExtraTTL | FELIX\_WINDOWSDNSEXTRATTL | Specify extra time in seconds to keep IPs and alias names that are learned from DNS, in addition to each name or IP's advertised TTL. | seconds | `120` | ### Kubernetes-specific configuration[​](#kubernetes-specific-configuration) | Configuration parameter | Environment variable | Description | Schema | | ----------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------ | | `KubeNodePortRanges` | `FELIX_KUBENODEPORTRANGES` | A list of port ranges that Felix should treat as Kubernetes node ports. Only when `kube-proxy` is configured to use IPVS mode: Felix assumes that traffic arriving at the host of one of these ports will ultimately be forwarded instead of being terminated by a host process. \[Default: `30000:32767`][]() | Comma-delimited list of `:` port ranges or single ports. | | `KubeMasqueradeBit` | `FELIX_KUBEMASQUERADEBIT` | KubeMasqueradeBit should be set to the same value as --iptables-masquerade-bit of kube-proxy when TPROXY is used. This defaults to the corresponding kube-proxy default value so it only needs to change if kube-proxy is using a non-standard setting. Must be within the range of 0-31. OpenShift sets the bit to 0 by default. \[Default: 14] | integer | > **SECONDARY:** []()When using Calico Cloud with Kubernetes' `kube-proxy` in IPVS mode, Calico Cloud uses additional iptables mark bits to store an ID for each local Calico Cloud endpoint. For example, the default `IptablesMarkMask` value, `0xffff0000` gives Calico Cloud 16 bits, up to 6 of which are used for internal purposes, leaving 10 bits for endpoint IDs. 10 bits is enough for 1024 different values and Calico Cloud uses 2 of those for internal purposes, leaving enough for 1022 endpoints on the host. ### Bare metal specific configuration[​](#bare-metal-specific-configuration) | Configuration parameter | Environment variable | Description | Schema | | ----------------------- | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | | `InterfacePrefix` | `FELIX_INTERFACEPREFIX` | The interface name prefix that identifies workload endpoints and so distinguishes them from host endpoint interfaces. Accepts more than one interface name prefix in comma-delimited format, e.g., `tap,cali`. Note: in environments other than bare metal, the orchestrators configure this appropriately. For example our Kubernetes and Docker integrations set the `cali` value, and our OpenStack integration sets the `tap` value. \[Default: `cali`] | string | ### Calico Cloud specific configuration[​](#calico-cloud-specific-configuration) | Setting | Environment variable | Default | Meaning | | --------------------------------------- | --------------------------------------------- | ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DropActionOverride` | `FELIX_DROPACTIONOVERRIDE` | `Drop` | How to treat packets that are disallowed by the current Calico Cloud policy. For more detail please see below. | | `LogDropActionOverride` | `FELIX_LOGDROPACTIONOVERRIDE` | `false` | Set to `true` to add the `DropActionOverride` to the syslog entries. For more detail please see below. | | `PrometheusReporterEnabled` | `FELIX_PROMETHEUSREPORTERENABLED` | `false` | Set to `true` to enable Prometheus reporting of denied packet metrics. For more detail please see below. | | `PrometheusReporterPort` | `FELIX_PROMETHEUSREPORTERPORT` | `9092` | The TCP port on which to report denied packet metrics. | | `PrometheusReporterCertFile` | `FELIX_PROMETHEUSREPORTERCERTFILE` | None | Certificate for encrypting Prometheus denied packet metrics. | | `PrometheusReporterKeyFile` | `FELIX_PROMETHEUSREPORTERKEYFILE` | None | Private key for encrypting Prometheus denied packet metrics. | | `PrometheusReporterCAFile` | `FELIX_PROMETHEUSREPORTERCAFILE` | None | Trusted CA file for clients attempting to read Prometheus denied packet metrics. | | `PrometheusMetricsCertFile` | `FELIX_PROMETHEUSMETRICSCERTFILE` | None | Certificate for encrypting general Felix Prometheus metrics. | | `PrometheusMetricsKeyFile` | `FELIX_PROMETHEUSMETRICSKEYFILE` | None | Private key for encrypting general Felix Prometheus metrics. | | `PrometheusMetricsCAFile` | `FELIX_PROMETHEUSMETRICSCAFILE` | None | Trusted CA file for clients attempting to read general Felix Prometheus metrics. | | `IPSecMode` | `FELIX_IPSECMODE` | None | Controls which mode IPsec is operating on. The only supported value is `PSK`. An empty value means IPsec is not enabled. | | `IPSecAllowUnsecuredTraffic` | `FELIX_IPSECALLOWUNSECUREDTRAFFIC` | `false` | When set to false, only IPsec-protected traffic will be allowed on the packet paths where IPsec is supported. When set to true, IPsec will be used but non-IPsec traffic will be accepted. In general, setting this to `true` is less safe since it allows an attacker to inject packets. However, it is useful when transitioning from non-IPsec to IPsec since it allows traffic to flow while the cluster negotiates the IPsec mesh. | | `IPSecIKEAlgorithm` | `FELIX_IPSECIKEALGORITHM` | `aes128gcm16-prfsha256-ecp256` | IPsec IKE algorithm. Default is NIST suite B recommendation. | | `IPSecESPAlgorithm` | `FELIX_IPSECESPALGORITHM` | `aes128gcm16-ecp256` | IPsec ESP algorithm. Default is NIST suite B recommendation. | | `IPSecLogLevel` | `FELIX_IPSECLOGLEVEL` | `Info` | Controls log level for IPsec components. Set to `None` for no logging. Other valid values are `Notice`, `Info`, `Debug` and `Verbose`. | | `IPSecPSKFile` | `FELIX_IPSECPSKFILE` | None | The path to the pre shared key file for IPsec. | | `FlowLogsFileEnabled` | `FELIX_FLOWLOGSFILEENABLED` | `false` | Set to `true`, enables flow logs. If set to `false` no flow logging will occur. Flow logs are written to a file `flows.log` and sent to Elasticsearch. The location of this file can be configured using the `FlowLogsFileDirectory` field. File rotation settings for this `flows.log` file can be configured using the fields `FlowLogsFileMaxFiles` and `FlowLogsFileMaxFileSizeMB`. Note that flow log exports to Elasticsearch are dependent on flow logs getting written to this file. Setting this parameter to `false` will disable flow logs. | | `FlowLogsFileIncludeLabels` | `FELIX_FLOWLOGSFILEINCLUDELABELS` | `false` | Set to `true` to include endpoint label information in flow logs. This parameter only takes effect when `FlowLogsFileEnabled` is set to `true`. | | `FlowLogsFileIncludePolicies` | `FELIX_FLOWLOGSFILEINCLUDEPOLICIES` | `false` | Set to `true` to include policy match information in flow logs. This parameter only takes effect when `FlowLogsFileEnabled` is set to `true`. | | `FlowLogsFileIncludeService` | `FELIX_FLOWLOGSFILEINCLUDESERVICE` | `false` | Set to `true` to include destination service information in flow logs. The service information is derived from pre-DNAT destination IP and is therefore only available on the node where DNAT occurs. This parameter only takes effect when `FlowLogsFileEnabled` is set to `true`. | | `FlowLogsFileDirectory` | `FELIX_FLOWLOGSFILEDIRECTORY` | `/var/log/calico/flowlogs` | The directory where flow logs files are stored. This parameter only takes effect when `FlowLogsFileEnabled` is set to `true`. | | `FlowLogsFileMaxFiles` | `FELIX_FLOWLOGSFILEMAXFILES` | `5` | The number of files to keep when rotating flow log files. This parameter only takes effect when `FlowLogsFileEnabled` is set to `true`. | | `FlowLogsFileMaxFileSizeMB` | `FELIX_FLOWLOGSFILEMAXFILESIZEMB` | `100` | The max size in MB of flow logs files before rotation. This parameter only takes effect when `FlowLogsFileEnabled` is set to `true`. | | `FlowLogsFlushInterval` | `FELIX_FLOWLOGSFLUSHINTERVAL` | `300` | The period, in seconds, at which Felix exports the flow logs. | | `FlowLogsEnableNetworkSets` | `FELIX_FLOWLOGSENABLENETWORKSETS` | `false` | Whether to specify the network set a flow log originates from. | | `FlowLogsFileAggregationKindForAllowed` | `FELIX_FLOWLOGSFILEAGGREGATIONKINDFORALLOWED` | `2` | How much to aggregate the flow logs sent to Elasticsearch for allowed traffic. Bear in mind that changing this value may have a dramatic impact on the volume of flow logs sent to Elasticsearch. `0` means no aggregation, `1` means aggregate all flows that share a source port on each node, `2` means aggregate all flows that share source ports or are from the same ReplicaSet and `3` means aggregate all flows that share destination and source ports and are from the same ReplicateSet | | `FlowLogsFileAggregationKindForDenied` | `FELIX_FLOWLOGSFILEAGGREGATIONKINDFORDENIED` | `1` | How much to aggregate the flow logs sent to Elasticsearch for denied traffic. Bear in mind that changing this value may have a dramatic impact on the volume of flow logs sent to Elasticsearch. `0` means no aggregation, `1` means aggregate all flows that share a source port on each node, and `2` means aggregate all flows that share source ports or are from the same ReplicaSet and `3` means aggregate all flows that share destination and source ports and are from the same ReplicateSet. | | `FlowLogsDynamicAggregationEnabled` | `FELIX_FLOWLOGSDYNAMICAGGREGATIONENABLED` | `false` | Enable dynamic aggregation for flow logs. This will increase aggregation up to the maximum level allowed (which is 3 and means aggregate all flows that share destination and source ports and are from the same ReplicateSet) when it detects the pipeline for reading flow logs is stalled. It will revert to its initial aggregation level when this condition changes. The initial aggregation level can be specified using `FlowLogsFileAggregationKindForAllowed` and `FlowLogsFileAggregationKindForDenied`. If these values are not specified, default values of `2` and `1` will be used. | | `FlowLogsPositionFilePath` | `FELIX_FLOWLOGSPOSITIONPATH` | `/var/log/calico/flows.log.pos` | Default path of the position file. It is used to read the current state of pipeline for flow logs. This parameter will be used only when `FlowLogsDynamicAggregationEnabled` is set to `true` | | `FlowLogsAggregationThresholdBytes` | `FELIX_FLOWLOGSAGGREGATIONTHRESHOLDBYTES` | `8192` | Default threshold to determine how far behind the pipeline for flow logs can get before aggregation starts in. Detecting a difference of 8192 bytes means increase 1 level, while a difference of 16384 means increasing two levels. This parameter will be used only when `FlowLogsDynamicAggregationEnabled` is set to `true`. | | `FlowLogsCollectProcessInfo` | `FELIX_FLOWLOGSCOLLECTPROCESSINFO` | `true` | If enabled Felix will load the kprobe BPF programs to collect process info. | | `FlowLogsCollectTcpStats` | `FELIX_FLOWLOGSCOLLECTTCPSTATS` | `true` | If enabled Felix will collect TCP socket stats using BPF and requires a recent kernel that supports BPF | | `FlowLogsCollectProcessPath` | `FELIX_FLOWLOGSCOLLECTPROCESSPATH` | `true` | If enabled, along with FlowLogsCollectProcessInfo, each flow log will contain the full path of the process executable and the arguments with which the executable was invoked. If path or arguments cannot be determined, Felix will fallback to using task names and arguments will be empty. For full functionality, this feature should be enabled via operator see [Enabling process path](https://docs.tigera.io/calico-cloud/observability/elastic/flow/processpath) | | `FlowLogsFilePerFlowProcessLimit` | `FELIX_FLOWLOGSFILEPERFLOWPROCESSLIMIT` | `2` | Specify the maximum number of flow log entries with distinct process information beyond which process information will be aggregated. | | `FlowLogsFilePerFlowProcessArgsLimit` | `FELIX_FLOWLOGSFILEPERFLOWPROCESSARGSLIMIT` | `5` | Specify the maximum number of arguments beyond which the process arguments will be aggregated. | | `FlowLogsPolicyEvaluationMode` | `FELIX_FLOWLOGSPOLICYEVALUATIONMODE` | `Continuous` | Defines how policies are evaluated and reflected in flow logs. `OnNewConnection`: In this mode, staged policies are only evaluated when new connections are made in the data plane. Staged/active policy changes will not be reflected in the `pending_policies` field of flow logs for long lived connections. `Continuous`: Felix evaluates active flows on a regular basis to determine the rule traces in the flow logs. Any policy updates that impact a flow will be reflected in the `pending_policies` field, offering a near-real-time view of policy changes across flows. | | `DNSCacheFile` | `FELIX_DNSCACHEFILE` | `/var/run/calico/felix-dns-cache.txt` | The name of the file that Felix uses to preserve learned DNS information when restarting. | | `DNSCacheSaveInterval` | `FELIX_DNSCACHESAVEINTERVAL` | `60` | The periodic interval at which Felix saves learned DNS information to the cache file. | | `DNSCacheEpoch` | `FELIX_DNSCACHEEPOCH` | `0` | An arbitrary number that can be changed, at runtime, to tell Felix to discard all its learned DNS information. | | `DNSExtraTTL` | `FELIX_DNSEXTRATTL` | `0` | Extra time, in seconds, to keep IPs and alias names that are learned from DNS, in addition to each name or IP's advertised TTL. | | `DNSTrustedServers` | `FELIX_DNSTRUSTEDSERVERS` | `k8s-service:kube-dns` | The DNS servers that Felix should trust. Each entry here must be `[:]` - indicating an explicit DNS server IP - or `k8s-service:[/][:port]` - indicating a Kubernetes DNS service. `` defaults to the first service port, or 53 for an IP, and `` to `kube-system`. An IPv6 address with a port must use the square brackets convention, for example `[fd00:83a6::12]:5353`. Note that Felix (calico-node) will need RBAC permission to read the details of each service specified by a `k8s-service:...` form. | | `DNSLogsFileEnabled` | `FELIX_DNSLOGSFILEENABLED` | `false` | Set to `true`, enables DNS logs. If set to `false` no DNS logging will occur. DNS logs are written to a file `dns.log` and sent to Elasticsearch. The location of this file can be configured using the `DNSLogsFileDirectory` field. File rotation settings for this `dns.log` file can be configured using the fields `DNSLogsFileMaxFiles` and `DNSLogsFileMaxFileSizeMB`. Note that DNS log exports to Elasticsearch are dependent on DNS logs getting written to this file. Setting this parameter to `false` will disable DNS logs. | | `DNSLogsFileDirectory` | `FELIX_DNSLOGSFILEDIRECTORY` | `/var/log/calico/dnslogs` | The directory where DNS logs files are stored. This parameter only takes effect when `DNSLogsFileEnabled` is `true`. | | `DNSLogsFileMaxFiles` | `FELIX_DNSLOGSFILEMAXFILES` | `5` | The number of files to keep when rotating DNS log files. This parameter only takes effect when `DNSLogsFileEnabled` is `true`. | | `DNSLogsFileMaxFileSizeMB` | `FELIX_DNSLOGSFILEMAXFILESIZEMB` | `100` | The max size in MB of DNS log files before rotation. This parameter only takes effect when `DNSLogsFileEnabled` is `true`. | | `DNSLogsFlushInterval` | `FELIX_DNSLOGSFLUSHINTERVAL` | `300` | The period, in seconds, at which Felix exports DNS logs. | | `DNSLogsFileAggregationKind` | `FELIX_DNSLOGSFILEAGGREGATIONKIND` | `1` | How much to aggregate DNS logs. Bear in mind that changing this value may have a dramatic impact on the volume of flow logs sent to Elasticsearch. `0` means no aggregation, `1` means aggregate similar DNS logs from workloads in the same ReplicaSet. | | `DNSLogsFileIncludeLabels` | `FELIX_DNSLOGSFILEINCLUDELABELS` | `true` | Whether to include client and server workload labels in DNS logs. | | `DNSLogsFilePerNodeLimit` | `FELIX_DNSLOGSFILEPERNODELIMIT` | `0` (no limit) | Limit on the number of DNS logs that can be emitted within each flush interval. When this limit has been reached, Felix counts the number of unloggable DNS responses within the flush interval, and emits a WARNING log with that count at the same time as it flushes the buffered DNS logs. | | `DNSLogsLatency` | `FELIX_DNSLOGSLATENCY` | `true` | Indicates to include measurements of DNS request/response latency in each DNS log. | | `EgressIPSupport` | `FELIX_EGRESSIPSUPPORT` | `Disabled` | Defines three different support modes for egress gateway function. `Disabled` means egress gateways are not supported. `EnabledPerNamespace` means egress gateway function is enabled and can be configured on a per-namespace basis (but per-pod egress annotations are ignored). `EnabledPerNamespaceOrPerPod` means egress gateway function is enabled and can be configured per-namespace or per-pod (with per-pod egress annotations overriding namespace annotations). | | `EgressIPVXLANPort` | `FELIX_EGRESSIPVXLANPORT` | `4097` | Port to use for egress gateway VXLAN traffic. A value of `0` means "use the kernel default". | | `EgressIPVXLANVNI` | `FELIX_EGRESSIPVXLANVNI` | `4790` | Virtual network ID to use for egress gateway VXLAN traffic. A value of `0` means "use the kernel default". | | `EgressIPRoutingRulePriority` | `FELIX_EGRESSIPROUTINGRULEPRIORITY` | `100` | Priority value to use for the egress gateway routing rule. | | `L7LogsFileEnabled` | `FELIX_L7LOGSFILEENABLED` | `true` | If set to `false` no L7 logging will occur. L7 logs are written to a file `l7.log` and sent to Elasticsearch. The location of this file can be configured using the `L7LogsFileDirectory` field. File rotation settings for this `l7.log` file can be configured using the fields `L7LogsFileMaxFiles` and `L7LogsFileMaxFileSizeMB`. Note that L7 log exports to Elasticsearch are dependent on L7 logs getting written to this file. | | `L7LogsFileDirectory` | `FELIX_L7LOGSFILEDIRECTORY` | `/var/log/calico/l7logs` | The directory where L7 log files are stored. This parameter only takes effect when `L7LogsFileEnabled` is `true`. | | `L7LogsFileMaxFiles` | `FELIX_L7LOGSFILEMAXFILES` | `5` | The number of files to keep when rotating L7 log files. This parameter only takes effect when `L7LogsFileEnabled` is `true`. | | `L7LogsFileMaxFileSizeMB` | `FELIX_L7LOGSFILEMAXFILESIZEMB` | `100` | The max size in MB of L7 log files before rotation. This parameter only takes effect when `L7LogsFileEnabled` is `true`. | | `L7LogsFlushInterval` | `FELIX_L7LOGSFLUSHINTERVAL` | `300` | The period, in seconds, at which Felix exports L7 logs. | | `L7LogsFileAggregationHTTPHeaderInfo` | `FELIX_L7LOGSFILEAGGREGATIONHTTPHEADERINFO` | `ExcludeL7HTTPHeaderInfo` | How to handle HTTP header information for aggregating L7 logs. Bear in mind that changing this value may have a dramatic impact on the volume of L7 logs sent to Elasticsearch. Possible values include `ExcludeL7HTTPHeaderInfo` and `IncludeL7HTTPHeaderInfo`. | | `L7LogsFileAggregationHTTPMethod` | `FELIX_L7LOGSFILEAGGREGATIONHTTPMETHOD` | `IncludeL7HTTPMethod` | How to handle HTTP method data for aggregating L7 logs. Bear in mind that changing this value may have a dramatic impact on the volume of L7 logs sent to Elasticsearch. Possible values include `ExcludeL7HTTPMethod` and `IncludeL7HTTPMethod`. | | `L7LogsFileAggregationServiceInfo` | `FELIX_L7LOGSFILEAGGREGATIONSERVICEINFO` | `IncludeL7ServiceInfo` | How to handle service information for aggregating L7 logs. Bear in mind that changing this value may have a dramatic impact on the volume of L7 logs sent to Elasticsearch. Possible values include `ExcludeL7ServiceInfo` and `IncludeL7ServiceInfo`. | | `L7LogsFileAggregationDestinationInfo` | `FELIX_L7LOGSFILEAGGREGATIONDESTINATIONINFO` | `IncludeL7DestinationInfo` | How to handle destination metadata for aggregating L7 logs. Bear in mind that changing this value may have a dramatic impact on the volume of L7 logs sent to Elasticsearch. Possible values include `ExcludeL7DestinationInfo` and `IncludeL7DestinationInfo`. | | `L7LogsFileAggregationSourceInfo` | `FELIX_L7LOGSFILEAGGREGATIONSOURCEINFO` | `IncludeL7SourceInfoNoPort` | How to handle source metadata for aggregating L7 logs. Bear in mind that changing this value may have a dramatic impact on the volume of L7 logs sent to Elasticsearch. Possible values include `ExcludeL7SourceInfo`, `IncludeL7SourceInfoNoPort`, and `IncludeL7SourceInfo`. | | `L7LogsFileAggregationResponseCode` | `FELIX_L7LOGSFILEAGGREGATIONRESPONSECODE` | `IncludeL7ResponseCode` | How to handle response code data for aggregating L7 logs. Bear in mind that changing this value may have a dramatic impact on the volume of L7 logs sent to Elasticsearch. Possible values include `ExcludeL7ResponseCode` and `IncludeL7ResponseCode`. | | `L7LogsFileAggregationTrimURL` | `FELIX_L7LOGSFILEAGGREGATIONTRIMURL` | `IncludeL7FullURL` | How to handle URL data for aggregating L7 logs. Bear in mind that changing this value may have a dramatic impact on the volume of L7 logs sent to Elasticsearch. Possible values include `ExcludeL7URL`, `TrimURLQuery`, `TrimURLQueryAndPath`, and `IncludeL7FullURL`. | | `L7LogsFileAggregationNumURLPath` | `FELIX_L7LOGSFILEAGGREGATIONNUMURLPATH` | `5` | How many components in the path to limit the URL by. This parameter only takes effect when `L7LogsFileAggregationTrimURL` is set to `IncludeL7FullURL`. Bear in mind that changing this value may have a dramatic impact on the volume of L7 logs sent to Elasticsearch. Negative values set the limit to infinity. | DropActionOverride controls what happens to each packet that is denied by the current Calico Cloud policy - i.e. by the ordered combination of all the configured policies and profiles that apply to that packet. It may be set to one of the following values: - `Drop` - `Accept` - `LogAndDrop` - `LogAndAccept` Normally the `Drop` or `LogAndDrop` value should be used, as dropping a packet is the obvious implication of that packet being denied. However when experimenting, or debugging a scenario that is not behaving as you expect, the `Accept` and `LogAndAccept` values can be useful: then the packet will be still be allowed through. When set to `LogAndDrop` or `LogAndAccept`, each denied packet is logged in syslog, with an entry like this: ```text May 18 18:42:44 ubuntu kernel: [ 1156.246182] calico-drop: IN=tunl0 OUT=cali76be879f658 MAC= SRC=192.168.128.30 DST=192.168.157.26 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=56743 DF PROTO=TCP SPT=56248 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0 MARK=0xa000000 ``` If the `LogDropActionOverride` flag is set, then the `DropActionOverride` will also appear in the syslog entry: ```text May 18 18:42:44 ubuntu kernel: [ 1156.246182] calico-drop LOGandDROP: IN=tunl0 OUT=cali76be879f658 MAC= SRC=192.168.128.30 DST=192.168.157.26 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=56743 DF PROTO=TCP SPT=56248 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0 MARK=0xa000000 ``` When the reporting of denied packet metrics is enabled, Felix keeps counts of recently denied packets and publishes these as Prometheus metrics on the port configured by the `PrometheusReporterPort` setting. Note that denied packet metrics are independent of the DropActionOverride setting. Specifically, if packets that would normally be denied are being allowed through by a setting of `Accept` or `LogAndAccept`, those packets still contribute to the denied packet metrics as just described. ### Felix-Typha Configuration[​](#felix-typha-configuration) | Configuration parameter | Environment variable | Description | Schema | | ----------------------- | --------------------------- | --------------------------------------------------------------------------------------------- | ------ | | `TyphaAddr` | `FELIX_TYPHAADDR` | Address of the Typha Server when running outside a K8S Cluster, in the format IP:PORT | string | | `TyphaK8sServiceName` | `FELIX_TYPHAK8SSERVICENAME` | Service Name of Typha Deployment when running inside a K8S Cluster | string | | `TyphaK8sNamespace` | `FELIX_TYPHAK8SNAMESPACE` | Namespace of Typha Deployment when running inside a K8S Cluster. \[Default: `kube-system`] | string | | `TyphaReadTimeout` | `FELIX_TYPHAREADTIMEOUT` | Timeout of Felix when reading information from Typha, in seconds. \[Default: 30] | int | | `TyphaWriteTimeout` | `FELIX_TYPHAWRITETIMEOUT` | Timeout of Felix when writing information to Typha, in seconds. \[Default: 30] | int | ### Felix-Typha TLS configuration[​](#felix-typha-tls-configuration) | Configuration parameter | Environment variable | Description | Schema | | ----------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------ | | `TyphaCAFile` | `FELIX_TYPHACAFILE` | Path to the file containing the root certificate of the CA that issued the Typha server certificate. Configures Felix to trust the CA that signed the root certificate. The file may contain multiple root certificates, causing Felix to trust each of the CAs included. Example: `/etc/felix/ca.pem` | string | | `TyphaCertFile` | `FELIX_TYPHACERTFILE` | Path to the file containing the client certificate issued to Felix. Enables Felix to participate in mutual TLS authentication and identify itself to the Typha server. Example: `/etc/felix/cert.pem` | string | | `TyphaCN` | `FELIX_TYPHACN` | If set, the `Common Name` that Typha's certificate must have. If you have enabled TLS on the communications from Felix to Typha, you must set a value here or in `TyphaURISAN`. You can set values in both, as well, such as to facilitate a migration from using one to the other. If either matches, the communication succeeds. \[Default: none] | string | | `TyphaKeyFile` | `FELIX_TYPHAKEYFILE` | Path to the file containing the private key matching the Felix client certificate. Enables Felix to participate in mutual TLS authentication and identify itself to the Typha server. Example: `/etc/felix/key.pem` (optional) | string | | `TyphaURISAN` | `FELIX_TYPHAURISAN` | If set, a URI SAN that Typha's certificate must have. We recommend populating this with a [SPIFFE](https://github.com/spiffe/spiffe/blob/master/standards/SPIFFE-ID.md#2-spiffe-identity) string that identifies Typha. All Typha instances should use the same SPIFFE ID. If you have enabled TLS on the communications from Felix to Typha, you must set a value here or in `TyphaCN`. You can set values in both, as well, such as to facilitate a migration from using one to the other. If either matches, the communication succeeds. \[Default: none] | string | For more information on how to use and set these variables, refer to [Connections from Node to Typha (Kubernetes)](https://docs.tigera.io/calico-cloud/operations/comms/crypto-auth#connections-from-node-to-typha-kubernetes). ### PacketCapture configuration[​](#packetcapture-configuration) The following parameters fine tune packet capture rotation: | Configuration parameter | Environment variable | Description | Schema | | ------------------------ | --------------------------------- | --------------------------------------------------------------------------------------------- | ------ | | `CaptureDir` | `FELIX_CAPTUREDIR` | Controls the directory where packet capture files are stored. Example: `/var/log/calico/pcap` | string | | `CaptureMaxSizeBytes` | `FELIX_CAPTUREMAXSIZEBYTES` | Controls the maximum size in bytes for a packet capture file before rotation. | int | | `CaptureRotationSeconds` | `FELIX_CAPTUREMAXROTATIONSECONDS` | Controls the rotation period in seconds for a packet capture file. | int | | `CaptureMaxFiles` | `FELIX_CAPTUREMAXFILES` | Controls the maximum number rotated packet capture files. | int | ### WireGuard configuration[​](#wireguard-configuration) | Configuration parameter | Environment variable | Description | Schema | | | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | ------- | -------------- | | wireguardEnabled | Enable encryption for IPv4 on WireGuard supported nodes in cluster. When enabled, pod to pod traffic will be sent over encrypted tunnels between the nodes. | `true`, `false` | boolean | `false` | | wireguardEnabledV6 | Enable encryption for IPv6 on WireGuard supported nodes in cluster. When enabled, pod to pod traffic will be sent over encrypted tunnels between the nodes. | `true`, `false` | boolean | `false` | | wireguardInterfaceName | Name of the IPv4 WireGuard interface created by Felix. If you change the name, and want to clean up the previously-configured interface names on each node, this is a manual process. | string | string | wireguard.cali | | wireguardInterfaceNameV6 | Name of the IPv6 WireGuard interface created by Felix. If you change the name, and want to clean up the previously-configured interface names on each node, this is a manual process. | string | string | wg-v6.cali | | wireguardListeningPort | Port used by IPv4 WireGuard tunnels. Felix sets up an IPv4 WireGuard tunnel on each node specified by this port. Available for configuration only in the global FelixConfiguration resource; setting it per host, config-file or environment variable will not work. | 1-65535 | int | 51820 | | wireguardListeningPortV6 | Port used by IPv6 WireGuard tunnels. Felix sets up an IPv6 WireGuard tunnel on each node specified by this port. Available for configuration only in the global FelixConfiguration resource; setting it per host, config-file or environment variable will not work. | 1-65535 | int | 51821 | | wireguardMTU | MTU set on the IPv4 WireGuard interface created by Felix. Zero value means auto-detect. See [Configuring MTU](https://docs.tigera.io/calico-cloud/networking/configuring/mtu). | int | int | 0 | | wireguardMTUV6 | MTU set on the IPv6 WireGuard interface created by Felix. Zero value means auto-detect. See [Configuring MTU](https://docs.tigera.io/calico-cloud/networking/configuring/mtu). | int | int | 0 | | wireguardRoutingRulePriority | WireGuard routing rule priority value set up by Felix. If you change the default value, set it to a value most appropriate to routing rules for your nodes. | 1-32765 | int | 99 | | wireguardHostEncryptionEnabled | **Experimental**: Adds host-namespace workload IP's to WireGuard's list of peers. Should **not** be enabled when WireGuard is enabled on a cluster's control plane node, as networking deadlock can occur. | true, false | boolean | false | | wireguardKeepAlive | WireguardKeepAlive controls Wireguard PersistentKeepalive option. Set 0 to disable. \[Default: 0] | int | int | 25 | For more information on encrypting in-cluster traffic with WireGuard, refer to [Encrypt cluster pod traffic](https://docs.tigera.io/calico-cloud/compliance/encrypt-cluster-pod-traffic) ## Environment variables[​](#environment-variables) The highest priority of configuration is that read from environment variables. To set a configuration parameter via an environment variable, set the environment variable formed by taking `FELIX_` and appending the uppercase form of the variable name. For example, to set the etcd address, set the environment variable `FELIX_ETCDADDR`. Other examples include `FELIX_ETCDSCHEME`, `FELIX_ETCDKEYFILE`, `FELIX_ETCDCERTFILE`, `FELIX_ETCDCAFILE`, `FELIX_FELIXHOSTNAME`, `FELIX_LOGFILEPATH` and `FELIX_METADATAADDR`. > **SECONDARY:** To set a parameter to an empty value using an environment variable, assign an empty string to it (e.g., `FELIX_FAILSAFEINBOUNDHOSTPORTS=""`). ## Configuration file[​](#configuration-file) On startup, Felix reads an ini-style configuration file. The path to this file defaults to `/etc/calico/felix.cfg` but can be overridden using the `-c` or `--config-file` options on the command line. If the file exists, then it is read (ignoring section names) and all parameters are set from it. In OpenStack, we recommend putting all configuration into configuration files, since the etcd database is transient (and may be recreated by the OpenStack plugin in certain error cases). However, in a Docker environment the use of environment variables or etcd is often more convenient. ## Datastore[​](#datastore) Felix also reads configuration parameters from the datastore. It supports a global setting and a per-host override. 1. Get the current felixconfig settings. ```bash kubectl get felixconfiguration.projectcalico.org default -o yaml --export > felix.yaml ``` 2. Modify logFilePath to your intended path, e.g. "/tmp/felix.log" ```bash vim felix.yaml ``` > **SUCCESS:** For a global change set name to "default". For a node-specific change: set name to `node.`, e.g. "node.Calico Cloud-node-1" 3. Replace the current felixconfig settings ```bash kubectl replace -f felix.yaml ``` For more information, see [Felix Configuration Resource](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig). ### Monitoring Felix with Prometheus Felix can be configured to report a number of metrics through Prometheus. See the [configuration reference](https://docs.tigera.io/calico-cloud/reference/component-resources/node/felix/configuration) for how to enable metrics reporting. ## Metric reference[​](#metric-reference) ### Felix specific[​](#felix-specific) Felix exports a number of Prometheus metrics. The current set is as follows. Since some metrics are tied to particular implementation choices inside Felix we can't make any hard guarantees that metrics will persist across releases. However, we aim not to make any spurious changes to existing metrics. | Name | Description | | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `felix_active_local_endpoints` | Number of active endpoints on this host. | | `felix_active_local_policies` | Number of active policies on this host. | | `felix_active_local_selectors` | Number of active selectors on this host. | | `felix_active_local_tags` | Number of active tags on this host. | | `felix_bpf_conntrack_cleaned` | Number of entries cleaned during a conntrack table sweep. | | `felix_bpf_conntrack_cleaned_total` | Total number of entries cleaned during conntrack table sweeps, incremented for each clean individually. | | `felix_bpf_conntrack_expired` | Number of entries cleaned during a conntrack table sweep due to expiration. | | `felix_bpf_conntrack_expired_total` | Total number of entries cleaned during conntrack table sweep due to expiration - by reason. | | `felix_bpf_conntrack_inforeader_blocks` | Conntrack InfoReader would-blocks. | | `felix_bpf_conntrack_stale_nat` | Number of entries cleaned during a conntrack table sweep due to stale NAT. | | `felix_bpf_conntrack_stale_nat_total` | Total number of entries cleaned during conntrack table sweeps due to stale NAT. | | `felix_bpf_conntrack_sweeps` | Number of conntrack table sweeps made so far. | | `felix_bpf_conntrack_used` | Number of used entries visited during a conntrack table sweep. | | `felix_bpf_conntrack_sweep_duration` | Conntrack sweep execution time (ns). | | `felix_bpf_num_ip_sets` | Number of BPF IP sets managed in the data plane. | | `felix_calc_graph_output_events` | Number of events emitted by the calculation graph. | | `felix_calc_graph_update_time_seconds` | Seconds to update calculation graph for each datastore OnUpdate call. | | `felix_calc_graph_updates_processed` | Number of datastore updates processed by the calculation graph. | | `felix_cluster_num_host_endpoints` | Total number of host endpoints cluster-wide. | | `felix_cluster_num_hosts` | Total number of Calico Cloud hosts in the cluster. | | `felix_cluster_num_policies` | Total number of policies in the cluster. | | `felix_cluster_num_profiles` | Total number of profiles in the cluster. | | `felix_cluster_num_tiers` | Total number of Calico Cloud tiers in the cluster. | | `felix_cluster_num_workload_endpoints` | Total number of workload endpoints cluster-wide. | | `felix_egress_gateway_remote_polls{status="total"}` | Total number of remote egress gateway pods that Felix is polling for health/connectivity. Only egress gateways with a named "health" port will be polled. | | `felix_egress_gateway_remote_polls{status="up"}` | Total number of remote egress gateway pods that have successful probes. | | `felix_egress_gateway_remote_polls{status="probe-failed"}` | Total number of remote egress gateway pods that have failed probes. | | `felix_exec_time_micros` | Summary of time taken to fork/exec child processes | | `felix_int_dataplane_addr_msg_batch_size` | Number of interface address messages processed in each batch. Higher values indicate we're doing more batching to try to keep up. | | `felix_int_dataplane_apply_time_seconds` | Time in seconds that it took to apply a data plane update. | | `felix_int_dataplane_failures` | Number of times data plane updates failed and will be retried. | | `felix_int_dataplane_iface_msg_batch_size` | Number of interface state messages processed in each batch. Higher values indicate we're doing more batching to try to keep up. | | `felix_int_dataplane_messages` | Number data plane messages by type. | | `felix_int_dataplane_msg_batch_size` | Number of messages processed in each batch. Higher values indicate we're doing more batching to try to keep up. | | `felix_ipsec_bindings_total` | Total number of ipsec bindings. | | `felix_ipsec_errors` | Number of ipsec command failures. | | `felix_ipset_calls` | Number of ipset commands executed. | | `felix_ipset_errors` | Number of ipset command failures. | | `felix_ipset_lines_executed` | Number of ipset operations executed. | | `felix_ipsets_calico` | Number of active Calico Cloud IP sets. | | `felix_ipsets_total` | Total number of active IP sets. | | `felix_iptables_chains` | Number of active iptables chains. | | `felix_iptables_lines_executed` | Number of iptables rule updates executed. | | `felix_iptables_lock_acquire_secs` | Time taken to acquire the iptables lock. | | `felix_iptables_lock_retries` | Number of times the iptables lock was already held and felix had to retry to acquire it. | | `felix_iptables_restore_calls` | Number of iptables-restore calls. | | `felix_iptables_restore_errors` | Number of iptables-restore errors. | | `felix_iptables_rules` | Number of active iptables rules. | | `felix_iptables_save_calls` | Number of iptables-save calls. | | `felix_iptables_save_errors` | Number of iptables-save errors. | | `felix_log_errors` | Number of errors encountered while logging. | | `felix_logs_dropped` | Number of logs dropped because the output stream was blocked. | | `felix_reporter_log_errors` | Number of errors encountered while logging in the Syslog. | | `felix_reporter_logs_dropped` | Number of logs dropped because the output was blocked in the Syslog reporter. | | `felix_resync_state` | Current datastore state. | | `felix_resyncs_started` | Number of times Felix has started resyncing with the datastore. | | `felix_route_table_list_seconds` | Time taken to list all the interfaces during a resync. | | `felix_route_table_per_iface_sync_seconds` | Time taken to sync each interface | Prometheus metrics are self-documenting, with metrics turned on, `curl` can be used to list the metrics along with their help text and type information. ```bash curl -s http://localhost:9091/metrics | head ``` Example response: ```text # HELP felix_active_local_endpoints Number of active endpoints on this host. # TYPE felix_active_local_endpoints gauge felix_active_local_endpoints 91 # HELP felix_active_local_policies Number of active policies on this host. # TYPE felix_active_local_policies gauge felix_active_local_policies 0 # HELP felix_active_local_selectors Number of active selectors on this host. # TYPE felix_active_local_selectors gauge felix_active_local_selectors 82 ... ``` ### Label indexing metrics[​](#label-indexing-metrics) The label index is a subcomponent of Felix that is responsible for calculating the set of endpoints and network sets that match each selector that is in an active policy rule. Policy rules are active on a particular node if the policy they belong to selects a workload or host endpoint on that node with its top-level selector (in `spec.selector`). Inactive policies have minimal CPU cost because their selectors do not get indexed. Since the label index must match the active selectors against *all* endpoints and network sets in the cluster, its performance is critical and it supports various optimizations to minimize CPU usage. Its metrics can be used to check that the optimizations are active for your policy set. #### `felix_label_index_num_endpoints`[​](#felix_label_index_num_endpoints) Reports the total number of endpoints (and similar objects such as network sets) being tracked by the index. This should match the number of endpoints and network sets in your cluster. #### `felix_label_index_num_active_selectors{optimized="true|false"}`[​](#felix_label_index_num_active_selectorsoptimizedtruefalse) Reports the total number of active selectors, broken into `optimized="true"` and `optimized="false"` sub-totals. The `optimized="true"` total tracks the number of selectors that the label index was able to optimize. Those selectors should be calculated efficiently even in clusters with hundreds of thousands of endpoints. In general the CPU used to calculate them should be proportional to the number of endpoints that match them and the churn rate of *those* endpoints. The `optimized="false"` total tracks the number of selectors that could not be optimized. Unoptimized selectors are much more costly to calculate; the CPU used to calculate them is proportional to the number of endpoints in the cluster and their churn rate. It is generally OK to have a handful of unoptimized selectors, but if many selectors are unoptimized the CPU usage can be substantial at high scale. For more information on writing selectors that can be optimized, see the [this](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy#selector-performance-in-entityrules) section of the `NetworkPolicy` reference. #### `felix_label_index_selector_evals{result="true|false"}`[​](#felix_label_index_selector_evalsresulttruefalse) Counts the total number of times that a selector was evaluated vs an endpoint to determine if it matches, broken down by match (`true`) or no-match (`false`). The ratio of match to no-match shows how effective the selector indexing optimizations are for your policy set. The more effectively the label index can optimize the selectors, the fewer "no-match" results it will report relative to "match". If you have more than a handful of active selectors and `felix_label_index_selector_evals{result="false"}` is many times `felix_label_index_selector_evals{result="true"}` then it is likely that some selectors in the policy set are not being optimized effectively. #### `felix_label_index_strategy_evals{strategy="..."}`[​](#felix_label_index_strategy_evalsstrategy) This is a technical statistic that shows how many times the label index has employed each optimization strategy that it has available. The strategies will likely evolve over time but, at time of writing, they are as follows: - `endpoint-full-scan`: the least efficient fall back strategy for unoptimized selectors. The index scanned *all* endpoints to find the matches for a selector. - `endpoint|parent-no-match`: the most efficient strategy; the index was able to prove that nothing matched the selector so it was able to skip the scan entirely. - `endpoint|parent-single-value`: the label index was able to limit the scan to only those endpoints/parents that have a particular label and value combination. For example, selector `label == "value"` would only scan items that had exactly that label set to "value". - `endpoint|parent-multi-value`: the label index was able to limit the scan to only those endpoints/parents that have a particular label and one of a few values. For example, selector `label in {"a", "b")` would only scan items that had exactly that label with one of the given values. - `endpoint|parent-label-name`: the label index was able to limit the scan to only those endpoints/parents that hava a particular label (but was unable to limit it to a particular subset of values). For example, `has(label)` would result in that kind of scan. Terminology: here "endpoint" means "endpoint or NetworkSet" and "parent" is Felix's internal name for resources like Kubernetes Namespaces. A "parent" scan means that the label index scanned all endpoints that have a parent matching the strategy. ### CPU / memory metrics[​](#cpu--memory-metrics) Felix also exports the default set of metrics that Prometheus makes available. Currently, those include: | Name | Description | | ---------------------------------- | ------------------------------------------------------------------------------------------- | | `go_gc_duration_seconds` | A summary of the GC invocation durations. | | `go_goroutines` | Number of goroutines that currently exist. | | `go_info` | Go version. | | `go_memstats_alloc_bytes` | Number of bytes allocated and still in use. | | `go_memstats_alloc_bytes_total` | Total number of bytes allocated, even if freed. | | `go_memstats_buck_hash_sys_bytes` | Number of bytes used by the profiling bucket hash table. | | `go_memstats_frees_total` | Total number of frees. | | `go_memstats_gc_cpu_fraction` | The fraction of this program’s available CPU time used by the GC since the program started. | | `go_memstats_gc_sys_bytes` | Number of bytes used for garbage collection system metadata. | | `go_memstats_heap_alloc_bytes` | Number of heap bytes allocated and still in use. | | `go_memstats_heap_idle_bytes` | Number of heap bytes waiting to be used. | | `go_memstats_heap_inuse_bytes` | Number of heap bytes that are in use. | | `go_memstats_heap_objects` | Number of allocated objects. | | `go_memstats_heap_released_bytes` | Number of heap bytes released to OS. | | `go_memstats_heap_sys_bytes` | Number of heap bytes obtained from system. | | `go_memstats_last_gc_time_seconds` | Number of seconds since 1970 of last garbage collection. | | `go_memstats_lookups_total` | Total number of pointer lookups. | | `go_memstats_mallocs_total` | Total number of mallocs. | | `go_memstats_mcache_inuse_bytes` | Number of bytes in use by mcache structures. | | `go_memstats_mcache_sys_bytes` | Number of bytes used for mcache structures obtained from system. | | `go_memstats_mspan_inuse_bytes` | Number of bytes in use by mspan structures. | | `go_memstats_mspan_sys_bytes` | Number of bytes used for mspan structures obtained from system. | | `go_memstats_next_gc_bytes` | Number of heap bytes when next garbage collection will take place. | | `go_memstats_other_sys_bytes` | Number of bytes used for other system allocations. | | `go_memstats_stack_inuse_bytes` | Number of bytes in use by the stack allocator. | | `go_memstats_stack_sys_bytes` | Number of bytes obtained from system for stack allocator. | | `go_memstats_sys_bytes` | Number of bytes obtained by system. Sum of all system allocations. | | `go_threads` | Number of OS threads created. | | `process_cpu_seconds_total` | Total user and system CPU time spent in seconds. | | `process_max_fds` | Maximum number of open file descriptors. | | `process_open_fds` | Number of open file descriptors. | | `process_resident_memory_bytes` | Resident memory size in bytes. | | `process_start_time_seconds` | Start time of the process since unix epoch in seconds. | | `process_virtual_memory_bytes` | Virtual memory size in bytes. | | `process_virtual_memory_max_bytes` | Maximum amount of virtual memory available in bytes. | ### Wireguard Metrics[​](#wireguard-metrics) Felix also exports wireguard device stats if found/detected. Can be disabled via Felix configuration. | Name | Description | | ------------------------------------ | ------------------------------------------------------------------------------------------------- | | `wireguard_meta` | Gauge. Device / interface information for a felix/calico node, values are in this metric's labels | | `wireguard_bytes_rcvd` | Counter. Current bytes received from a peer identified by a peer public key and endpoint | | `wireguard_bytes_sent` | Counter. Current bytes sent to a peer identified by a peer public key and endpoint | | `wireguard_latest_handshake_seconds` | Gauge. Last handshake with a peer, unix timestamp in seconds. | ### Configuration on public clouds ## [📄️ Amazon Web Services](https://docs.tigera.io/calico-cloud/reference/public-cloud/aws) [Advantages of using Calico Cloud in AWS.](https://docs.tigera.io/calico-cloud/reference/public-cloud/aws) ## [📄️ Azure](https://docs.tigera.io/calico-cloud/reference/public-cloud/azure) [Support for Calico Cloud in Azure.](https://docs.tigera.io/calico-cloud/reference/public-cloud/azure) ## [📄️ Google Compute Engine](https://docs.tigera.io/calico-cloud/reference/public-cloud/gce) [Methods to ensure that traffic between containers on different hosts is not dropped by GCE fabric.](https://docs.tigera.io/calico-cloud/reference/public-cloud/gce) ### Amazon Web Services Calico Cloud provides the following advantages when running in Amazon Web Services (AWS): - **Network Policy for Containers**: Calico Cloud provides fine-grained network security policy for individual containers. - **No Overlays**: Within each VPC subnet Calico Cloud doesn't need an overlay, which means high performance networking for your containers. - **No 50 Node Limit**: Calico Cloud allows you to surpass the 50 node limit, which exists as a consequence of the [AWS 50 route limit](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_Limits.html#vpc-limits-route-tables) when using the VPC routing table. ## Routing traffic within a single VPC subnet[​](#routing-traffic-within-a-single-vpc-subnet) Since Calico Cloud assigns IP addresses outside the range used by AWS for EC2 instances, you must disable AWS src/dst checks on each EC2 instance in your cluster [as described in the AWS documentation](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_NAT_Instance.html#EIP_Disable_SrcDestCheck). This allows Calico Cloud to route traffic natively within a single VPC subnet without using an overlay or any of the limited VPC routing table entries. ## Routing traffic across different VPC subnets / VPCs[​](#routing-traffic-across-different-vpc-subnets--vpcs) If you need to split your deployment across multiple AZs for high availability then each AZ will have its own VPC subnet. To use Calico Cloud across multiple different VPC subnets or [peered VPCs](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-peering.html), in addition to disabling src/dst checks as described above you must also enable IPIP encapsulation and outgoing NAT on your Calico Cloud IP pools. See the [IP pool configuration reference](https://docs.tigera.io/calico-cloud/reference/resources/ippool) for information on how to configure Calico Cloud IP pools. By default, Calico Cloud's IPIP encapsulation applies to all container-to-container traffic. However, encapsulation is only required for container traffic that crosses a VPC subnet boundary. For better performance, you can configure Calico Cloud to perform IPIP encapsulation only across VPC subnet boundaries. To enable the "CrossSubnet" IPIP feature, configure your Calico Cloud IP pool resources to enable IPIP and set the mode to "CrossSubnet". > **SECONDARY:** This feature was introduced in Calico Cloud v2.1, if your deployment was created with an older version of Calico Cloud, or if you if you are unsure whether your deployment is configured correctly, follow the [Configuring IP-in-IP guide](https://docs.tigera.io/calico-cloud/networking/configuring/vxlan-ipip) which discusses this in more detail. The following `kubectl` command will create or modify an IPv4 pool with CIDR 192.168.0.0/16 using IPIP mode `CrossSubnet`. Adjust the pool CIDR for your deployment. ```bash kubectl apply -f - < **WARNING:** Required for [federation](https://docs.tigera.io/calico-cloud/multicluster/overview). To disable SNAT on external packets, open the AWS VPC CNI manifest in your favorite editor and add an `AWS_VPC_K8S_CNI_EXTERNALSNAT` environment variable set to `true` in the `aws-node` container. An example follows. ```yaml kind: DaemonSet apiVersion: apps/v1 # kubernetes versions before 1.9.0 should use extensions/v1beta1 metadata: name: aws-node namespace: kube-system labels: k8s-app: aws-node spec: updateStrategy: type: RollingUpdate selector: matchLabels: k8s-app: aws-node template: metadata: labels: k8s-app: aws-node annotations: scheduler.alpha.kubernetes.io/critical-pod: '' spec: serviceAccountName: aws-node hostNetwork: true tolerations: - operator: Exists containers: - image: 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.3.4 imagePullPolicy: Always ports: - containerPort: 61678 name: metrics name: aws-node env: - name: AWS_VPC_K8S_CNI_LOGLEVEL value: DEBUG - name: MY_NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: AWS_VPC_K8S_CNI_EXTERNALSNAT value: 'true' ``` > **SECONDARY:** For details see the [Amazon VPC CNI Plugin Version 1.1](https://aws.amazon.com/blogs/opensource/vpc-cni-plugin-v1-1-available) release notes. 3. Apply the manifest using kubectl. ```bash kubectl apply -f aws-k8s-cni.yaml ``` 4. Follow the instructions to install Calico Cloud on AWS. ### Azure ## About Calico Cloud on Azure[​](#about-calico-cloud-on-azure) You can use Calico Cloud policy with one of the following networking options. - **Azure user-defined routes**: This option provides networking without overlays. Disable Calico Cloud networking by setting `CALICO_NETWORKING_BACKEND` to `none` in `cnx-node`. (Also called "policy-only mode".) Refer to [Configuring cnx-node](https://docs.tigera.io/calico-cloud/reference/component-resources/node/configuration) and [Azure user-defined routes](#azure-user-defined-routes) for more information. - **Calico VXLAN**: Install Calico Cloud using VXLAN encapsulation for pod traffic. - **Azure CNI IPAM plug-in**: Configure Calico Cloud to use the [Azure CNI plug-in](https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md) instead of the Calico Cloud CNI plug-in. ## Azure user-defined routes[​](#azure-user-defined-routes) To configure Azure user-defined routes (UDR): - Create an [Azure route table](https://docs.microsoft.com/en-us/azure/virtual-network/create-user-defined-route-portal) and associate it with the VMs subnet. - Enable [IP forwarding enabled](https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-network-interface#enable-or-disable-ip-forwarding) in your VM network interfaces. On Kubernetes, also complete the following. - Ensure that the selected pod's subnet is a part of your Azure virtual network IP range. - Include the name of your routing table in the configuration file of your Kubernetes Azure cloud provider. ## Does Azure support Calico Cloud networking?[​](#does-azure-support-calico-cloud-networking) Calico in VXLAN mode is supported on Azure. However, IPIP packets are blocked by the Azure network fabric. ### Google Compute Engine To deploy Calico Cloud in [Google Compute Engine (GCE)](https://cloud.google.com/compute/), you must ensure that traffic between containers on different hosts is not dropped by the GCE fabric. There are a few different options for doing this depending on your deployment. ## IP-in-IP encapsulation[​](#ip-in-ip-encapsulation) Container traffic routing can be enabled by setting [IP-in-IP encapsulation](https://docs.tigera.io/calico-cloud/networking/configuring/vxlan-ipip) and NAT outgoing on the configured Calico Cloud IP pools. See the [IP pool configuration reference](https://docs.tigera.io/calico-cloud/reference/resources/ippool) for information on how to configure Calico Cloud IP pools. ## GCE cloud routes[​](#gce-cloud-routes) Traffic routing in GCE can be achieved by utilizing GCE cloud routes and running Calico Cloud in policy-only mode. Kubernetes GCE cloud provider integration simplifies route configuration by enabling Kubernetes to handle creating routes. ## Enabling workload-to-WAN traffic[​](#enabling-workload-to-wan-traffic) To allow Calico Cloud networked containers to reach resources outside of GCE, you must configure outgoing NAT on your [Calico Cloud IP pool](https://docs.tigera.io/calico-cloud/reference/resources/ippool). ### Host endpoints ## [📄️ Host endpoints](https://docs.tigera.io/calico-cloud/reference/host-endpoints/overview) [Secure host network interfaces.](https://docs.tigera.io/calico-cloud/reference/host-endpoints/overview) ## [📄️ Creating policy for basic connectivity](https://docs.tigera.io/calico-cloud/reference/host-endpoints/connectivity) [Customize the Calico failsafe policy to protect host endpoints.](https://docs.tigera.io/calico-cloud/reference/host-endpoints/connectivity) ## [📄️ Creating host endpoint objects](https://docs.tigera.io/calico-cloud/reference/host-endpoints/objects) [To protect a host interface, start by creating a host endpoint object in etcd.](https://docs.tigera.io/calico-cloud/reference/host-endpoints/objects) ## [📄️ Selector-based policies](https://docs.tigera.io/calico-cloud/reference/host-endpoints/selector) [Apply ordered policies to endpoints that match specific label selectors.](https://docs.tigera.io/calico-cloud/reference/host-endpoints/selector) ## [📄️ Failsafe rules](https://docs.tigera.io/calico-cloud/reference/host-endpoints/failsafe) [Avoid cutting off connectivity to hosts because of incorrect network policies.](https://docs.tigera.io/calico-cloud/reference/host-endpoints/failsafe) ## [📄️ Pre-DNAT policy](https://docs.tigera.io/calico-cloud/reference/host-endpoints/pre-dnat) [Apply rules in a host endpoint policy before any DNAT.](https://docs.tigera.io/calico-cloud/reference/host-endpoints/pre-dnat) ## [📄️ Apply on forwarded traffic](https://docs.tigera.io/calico-cloud/reference/host-endpoints/forwarded) [Learn the subtleties using the applyOnForward option in host endpoint policies.](https://docs.tigera.io/calico-cloud/reference/host-endpoints/forwarded) ## [📄️ Summary of host endpoint policies](https://docs.tigera.io/calico-cloud/reference/host-endpoints/summary) [How different host endpoint rules affect packet flows.](https://docs.tigera.io/calico-cloud/reference/host-endpoints/summary) ## [📄️ Connection tracking](https://docs.tigera.io/calico-cloud/reference/host-endpoints/conntrack) [Workaround for Linux conntrack if Calico policy is not working as it should.](https://docs.tigera.io/calico-cloud/reference/host-endpoints/conntrack) ### Host endpoints This guide describes how to use Calico Cloud to secure the network interfaces of the host itself (as opposed to those of any container/VM workloads that are present on the host). We call such interfaces "host endpoints", to distinguish them from "workload endpoints" (such as containers or VMs). Calico Cloud supports the same rich security policy model for host endpoints (host endpoint policy) that it supports for workload endpoints. Host endpoints can have labels, and their labels are in the same "namespace" as those of workload endpoints. This allows security rules for either type of endpoint to refer to the other type (or a mix of the two) using labels and selectors. Calico Cloud does not support setting IPs or policing MAC addresses for host interfaces, it assumes that the interfaces are configured by the underlying network fabric. Calico Cloud distinguishes workload endpoints from host endpoints by a configurable prefix. Unless you happen to have host interfaces whose name matches the default for that prefix (`cali`), you won't need to change it. In case you do, see the `InterfacePrefix` configuration value at [Configuring Felix](https://docs.tigera.io/calico-cloud/reference/component-resources/node/felix/configuration) . Interfaces that start with a value listed in `InterfacePrefix` are assumed to be workload interfaces. Others are treated as host interfaces. Calico Cloud blocks all traffic to/from workload interfaces by default; allowing traffic only if the interface is known and policy is in place. However, for host endpoints, Calico Cloud is more lenient; it only polices traffic to/from interfaces that it's been explicitly told about. Traffic to/from other interfaces is left alone. You can use host endpoint policy to secure a NAT gateway or router. Calico Cloud supports selector-based policy when running on a gateway or router, allowing for rich, dynamic security policy based on the labels attached to your host endpoints. You can apply host endpoint policies to three types of traffic: - Traffic that is terminated locally. - Traffic that is forwarded between host endpoints. - Traffic that is forwarded between a host endpoint and a workload endpoint on the same host. Set the `applyOnForward` flag to `true` to apply a policy to forwarded traffic. See [GlobalNetworkPolicy spec](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy#spec). > **SECONDARY:** Both traffic forwarded between host endpoints and traffic forwarded between a host endpoint and a workload endpoint on the same host is regarded as `forwarded traffic`. ![Diagram showing packet flows for forwarded traffic between host endpoints and between host endpoints and workload endpoints](https://docs.tigera.io/assets/images/bare-metal-packet-flows-ddcd712692d4728941d8d204ebbb7e60.svg) ### Creating policy for basic connectivity When a host endpoint is added, if there is no security policy for that endpoint, Calico Cloud will default to denying traffic to/from that endpoint, except for traffic that is allowed by the [failsafe rules](https://docs.tigera.io/calico-cloud/reference/host-endpoints/failsafe). While the [failsafe rules](https://docs.tigera.io/calico-cloud/reference/host-endpoints/failsafe) provide protection against removing all connectivity to a host: - They are overly broad in allowing inbound SSH on any interface and allowing traffic out to etcd's ports on any interface. - Depending on your network, they may not cover all the ports that are required; for example, your network may rely on allowing ICMP, or DHCP. Therefore, we recommend creating a failsafe Calico Cloud security policy that is tailored to your environment. The example command below shows one example of how you might do that; the command uses `calicoctl` to create a single policy resource, which: - Applies to all known endpoints. - Allows inbound ssh access from a defined “management” subnet. - Allows outbound connectivity to etcd on a particular IP; if you have multiple etcd servers you should duplicate the rule for each destination. - Allows inbound ICMP. - Allows outbound UDP on port 67, for DHCP. When running this command, replace the placeholders in angle brackets with appropriate values for your deployment. ```bash cat <" destination: ports: [22] - action: Allow protocol: ICMP egress: - action: Allow protocol: TCP destination: nets: [/32] ports: [] - action: Allow protocol: TCP destination: nets: [] - action: Allow protocol: UDP destination: ports: [67] EOF ``` Once you have such a policy in place, you may want to disable the [failsafe rules](https://docs.tigera.io/calico-cloud/reference/host-endpoints/failsafe). > **SECONDARY:** Packets that reach the end of the list of rules fall-through to the next policy (sorted by the `order` field). The selector in the policy, `all()`, will match *all* endpoints, including any workload endpoints. If you have workload endpoints as well as host endpoints then you may wish to use a more restrictive selector. For example, you could label management interfaces with label `endpoint_type = management` and then use selector `endpoint_type == "management"` If you are using Calico Cloud for networking workloads, you should add inbound and outbound rules to allow BGP: add an ingress and egress rule to allow TCP traffic to destination port 179. ### Creating host endpoint objects For each host endpoint that you want Calico Cloud to secure, you'll need to create a host endpoint object in etcd. Use the `calicoctl create` command to create a host endpoint resource (`HostEndpoint`). There are two ways to specify the interface that a host endpoint should refer to. You can either specify the name of the interface or its expected IP address. In either case, you'll also need to know the name given to the Calico Cloud node running on the host that owns the interface; in most cases this will be the same as the hostname of the host. For example, to secure the interface named `eth0` with IP 10.0.0.1 on host `my-host`, run the command below. The name of the endpoint is an arbitrary name required for endpoint identification. When running this command, replace the placeholders in angle brackets with appropriate values for your deployment. ```bash calicoctl create -f - < labels: role: webserver environment: production spec: interfaceName: eth0 node: profiles: [] expectedIPs: ["10.0.0.1"] EOF ``` > **SECONDARY:** Felix tries to detect the correct hostname for a system. It logs out the value it has determined at start-of-day in the following format: `2015-10-20 17:42:09,813 \[INFO\]\[30149/5\] calico.felix.config 285: Parameter FelixHostname (Felix compute host hostname) has value 'my-hostname' read from None` The value (in this case `'my-hostname'`) needs to match the hostname used in etcd. Ideally, the host's system hostname should be set correctly but if that's not possible, the Felix value can be overridden with the FelixHostname configuration setting. See configuration for more details. Where `` is an optional list of security profiles to apply to the endpoint and labels contains a set of arbitrary key/value pairs that can be used in selector expressions. > **SECONDARY:** When rendering security rules on other hosts, Calico Cloud uses the `expectedIPs` field to resolve label selectors to IP addresses. If the `expectedIPs` field is omitted then security rules that use labels will fail to match this endpoint. Or, if you knew that the IP address should be 10.0.0.1, but not the name of the interface: ```bash calicoctl create -f - < labels: role: webserver environment: production spec: node: profiles: [] expectedIPs: ["10.0.0.1"] EOF ``` After you create host endpoint objects, Felix will start policing traffic to/from that interface. If you have no policy or profiles in place, then you should see traffic being dropped on the interface. > **SECONDARY:** By default, Calico Cloud has a failsafe in place that allows certain traffic such as ssh. See below for more details on disabling/configuring the failsafe rules. If you don't see traffic being dropped, check the hostname, IP address and (if used) the interface name in the configuration. If there was something wrong with the endpoint data, Felix will log a validation error at `WARNING` level and it will ignore the endpoint: A `grep` through the Felix logs for the string "Validation failed" should allow you to locate the error. ```bash grep "Validation failed" /var/log/calico/felix.log ``` An example error follows. ```text 2016-05-31 12:16:21,651 [WARNING][8657/3] calico.felix.fetcd 1017: Validation failed for host endpoint HostEndpointId, treating as missing: 'name' or 'expected_ipvX_addrs' must be present.; '{ "labels": {"foo": "bar"}, "profile_ids": ["prof1"]}' ``` The error can be quite long but it should log the precise cause of the rejection; in this case `'name' or 'expected\_ipvX\_addrs' must be present` tells us that either the interface's name or its expected IP address must be specified. ### Selector-based policies We recommend using selector-based security policy with host endpoints. This allows ordered policy to be applied to endpoints that match particular label selectors. For example, you could add a second policy for webserver access: ```bash cat < **SECONDARY:** Removing the inbound failsafe rules can leave a host inaccessible. > > Removing the outbound failsafe rules can leave Felix unable to connect to the datastore. > > Before disabling the failsafe rules, we recommend creating a policy to replace it with more-specific rules for your environment: see [Creating policy for basic connectivity](https://docs.tigera.io/calico-cloud/reference/host-endpoints/connectivity). ### Pre-DNAT policy Policy for host endpoints can be marked as `preDNAT`. This means that rules in that policy should be applied before any DNAT (Destination Network Address Translation), which is useful if it is more convenient to specify Calico Cloud policy in terms of a packet's original destination IP address and port, than in terms of that packet's destination IP address and port after it has been DNAT'd. An example is securing access to Kubernetes NodePorts from outside the cluster. Traffic from outside is addressed to any node's IP address, on a known NodePort, and Kubernetes (kube-proxy) then DNATs that to the IP address of one of the pods that provides the corresponding service, and the relevant port number on that pod (which is usually different from the NodePort). As NodePorts are the externally advertised way of connecting to services (and a NodePort uniquely identifies a service, whereas an internal port number may not), it makes sense to express Calico Cloud policy to expose or secure particular Services in terms of the corresponding NodePorts. But that is only possible if the Calico Cloud policy is applied before DNAT changes the NodePort to something else. Hence this kind of policy needs `preDNAT` set to `true`. In addition to being applied before any DNAT, the enforcement of pre-DNAT policy differs from that of normal host endpoint policy in three key details, reflecting that it is designed for the policing of incoming traffic from outside the cluster: - Pre-DNAT policy may only have ingress rules, not egress. (When incoming traffic is allowed by the ingress rules, standard connection tracking is sufficient to allow the return path traffic.) - Pre-DNAT policy is enforced for all traffic arriving through a host endpoint, regardless of where that traffic is going, and - in particular - even if that traffic is routed to a local workload on the same host. (Whereas normal host endpoint policy is skipped, for traffic going to a local workload.) - There is no 'default drop' semantic for pre-DNAT policy (as there is for normal host endpoint policy). In other words, if a host endpoint is defined but has no pre-DNAT policies that explicitly allow or deny a particular incoming packet, that packet is allowed to continue on its way, and will then be accepted or dropped according to workload policy (if it is going to a local workload) or to normal host endpoint policy (if not). ### Apply on forwarded traffic If `applyOnForward` is `false`, the host endpoint policy applies to traffic to/from local processes only. If `applyOnForward` is `true`, the host endpoint policy also applies to forwarded traffic: - Traffic that comes in via a host endpoint and is forwarded to a local workload (container/pod/VM). - Traffic from a local workload that is forwarded out via a host endpoint. - Traffic that comes in via a host endpoint and is forwarded out via another host endpoint. By default, `applyOnForward` is `false`. Untracked policies and pre-DNAT policies must have `applyOnForward` set to `true` because they apply to all forwarded traffic. Forwarded traffic is allowed by default if no policies apply to the endpoint and direction. In other words, if a host endpoint is configured, but there are no policies with `applyOnForward` set to `true` that apply to that host endpoint and traffic direction, forwarded traffic is allowed in that direction. For example if a forwarded flow is incoming via a host endpoint, but there are no Ingress policies with `applyOnForward: true` that apply to that host endpoint, the flow is allowed. If there are `applyOnForward: true` policies that select the host endpoint and direction, but no rules in the policies allow the traffic, the traffic is denied. This is different from how Calico Cloud treats traffic to or from a local process: if a host endpoint is configured and there are no policies that select the host endpoint in the traffic direction, or no rules that allow the traffic, the traffic is denied. Traffic that traverses a host endpoint and is forwarded to a workload endpoint must also pass the applicable workload endpoint policy, if any. That is to say, if an `applyOnForward: true` host endpoint policy allows the traffic, but workload endpoint policy denies it, the packet is still dropped. Traffic that ingresses one host endpoint, is forwarded, and egresses host endpoint must pass ingress policy on the first host endpoint and egress policy on the second host endpoint. > **SECONDARY:** Calico Cloud's handling of host endpoint policy has changed, since before Calico v3.0, in two ways: > > - It will not apply at all to forwarded traffic, by default. If you have an existing policy and you want it to apply to forwarded traffic, you need to add `applyOnForward: true` to the policy. > - Even with `applyOnForward: true`, the treatment is not quite the same in Calico v3.0 as in previous releases, because–once a host endpoint is configured– Calico v3.0 allows forwarded traffic through that endpoint by default, whereas previous releases denied forwarded traffic through that endpoint by default. If you want to maintain the default-deny behavior for all host-endpoint forwarded traffic, you can create an empty policy with `applyOnForward` set to `true` that applies to all traffic on all host endpoints. ```bash calicoctl apply -f - < **SECONDARY:** This policy has no `order` field specified which causes it to default to the highest value. Because higher order values have the lowest order of precedence, Calico Cloud will apply this policy after all other policies. Refer to the [policy spec](https://docs.tigera.io/calico-cloud/reference/resources/networkpolicy#spec) for more discussion. ### Summary of host endpoint policies Normal host endpoint policies apply to traffic that arrives on and/or is sent to a host interface, but the rules for applying untracked and pre-DNAT policies differ in some cases. Here we present and summarize all of those rules together, for all possible flows and all types of host endpoints policy. For packets that arrive on a host interface and are destined for a local workload, i.e., a locally-hosted pod, container or VM: - Pre-DNAT policies apply. - Normal policies do apply if `applyOnForward` is `true`. Normal policies do not apply if `applyOnForward` is `false`. - Untracked policies technically do apply, but never have any net positive effect for such flows. > **SECONDARY:** To be precise, untracked policy for the incoming host interface may apply in the forwards direction, and if so it will have the effect of forwarding the packet to the workload without any connection tracking. But then, in the reverse direction, there will be no conntrack state for the return packets to match, and there is no application of any egress rules that may be defined by the untracked policy—so unless the workload's policy specifically allows the relevant source IP, the return packet will be dropped. That is the same overall result as if there was no untracked policy at all, so in practice it is as if untracked policies do not apply to this flow. For packets that arrive on a host interface and are destined for a local server process in the host namespace: - Untracked, pre-DNAT and normal policies all apply. - If a packet is explicitly allowed by untracked policy, it skips over any pre-DNAT and normal policy. - If a packet is explicitly allowed by pre-DNAT policy, it skips over any normal policy. For packets that arrive on a host interface (A) and are forwarded out of the same or another host interface (B): - Untracked policies apply, for both host interfaces A and B, but only the ingress rules that are defined in those policies. The forwards direction is governed by the ingress rules of untracked policies that apply to interface A, and the reverse direction is governed by the ingress rules of untracked policies that apply to interface B, so those rules should be defined symmetrically. - Pre-DNAT policies apply, specifically the ingress rules of the pre-DNAT policies that apply to interface A. (The reverse direction is allowed by conntrack state.) - Normal policies apply if `applyOnForward` is `true`: specifically, the ingress rules of the normal policies that apply to interface A, and the egress rules of the normal policies that apply to interface B. (The reverse direction is allowed by conntrack state.) Normal policies do not apply if `applyOnForward` is `false`. - If a packet is explicitly allowed by untracked policy, it skips over any pre-DNAT and normal policy. - If a packet is explicitly allowed by pre-DNAT policy, it skips over any normal policy. For packets that are sent from a local server process (in the host namespace) out of a host interface: - Untracked policies apply, specifically the egress rules of the untracked policies that apply to the host interface. - Normal policies apply, specifically the egress rules of the normal policies that apply to that host interface. - Pre-DNAT policies do not apply. For packets that are sent from a local workload out of a host interface: - No untracked or pre-DNAT host endpoint policies apply. - Normal policies apply if `applyOnForward` is `true`: specifically, the egress rules of the normal policies that apply to the outgoing interface. (The reverse direction is allowed by conntrack state.) Normal policies do not apply if `applyOnForward` is `false`. ### Connection tracking Calico Cloud uses Linux's connection tracking ('conntrack') as an important optimization to its processing. It generally means that Calico Cloud only needs to check its policies for the first packet in an allowed flow—between a pair of IP addresses and ports—and then conntrack automatically allows further packets in the same flow, without Calico Cloud rechecking every packet. This can, however, make it look like a Calico Cloud policy is not working as it should, if policy is changed to disallow a flow that was previously allowed. If packets were recently exchanged on the previously allowed flow, and so there is conntrack state for that flow that has not yet expired, that conntrack state will allow further packets between the same IP addresses and ports, even after the Calico Cloud policy has been changed. Per Calico Cloud's current implementation, there are two workarounds for this: - Somehow ensure that no further packets flow between the relevant IP addresses and ports until the conntrack state has expired (typically about a minute). - Use the 'conntrack' tool to delete the relevant conntrack state; for example `conntrack -D -p tcp --orig-port-dst 80`. Then you should observe that the new Calico Cloud policy is enforced for new packets. ### Attribution ## Calico Cloud attribution report[​](#calico-cloud-attribution-report) 24 Feb 2021 Calico Cloud incorporates various open source softwares. The following open source components and their respective licenses used in the product are provided for your informational purposes. In the table below, you can look at the details of each project and license associated with it. | Project | Artifact Name | LICENSE | | ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------- | -------------------- | | abbrev | abbrev-1.1.1.tgz | ISC | | algoliasearch | algoliasearch.umd-4.2.0.js | MIT | | ansi-regex | ansi-regex-2.1.1.tgz | MIT | | ansi-styles | ansi-styles-3.2.1.tgz | MIT | | @types/anymatch | anymatch-1.3.1.tgz | MIT | | app-policy | app-policy-v3.16.6 | Tigera Proprietary | | argparse | argparse-1.0.9.tgz | MIT | | arr-diff | arr-diff-4.0.0.tgz | MIT | | arr-flatten | arr-flatten-1.1.0.tgz | MIT | | arr-union | arr-union-3.1.0.tgz | MIT | | array-unique | array-unique-0.3.2.tgz | MIT | | asap | asap-2.0.6.tgz | MIT | | assign-symbols | assign-symbols-1.0.0.tgz | MIT | | atob | atob-2.1.2.tgz | Apache 2.0 | | atomicwrites | atomicwrites-1.4.0-py2.py3-none-any.whl | MIT | | attrs | attrs-20.3.0-py2.py3-none-any.whl | MIT | | avsdf-base | avsdf-base-1.0.0.tgz | MIT | | babel-standalone | babel-6.26.0.min.js | MIT | | babel-runtime | babel-runtime-6.26.0.tgz | MIT | | backports.functools-lru-cache | backports.functools\_lru\_cache-1.6.1-py2.py3-none-any.whl | MIT | | balanced-match | balanced-match-0.4.2.tgz | MIT | | balanced-match | balanced-match-1.0.0.tgz | MIT | | base | base-0.11.2.tgz | MIT | | beautifulsoup4 | beautifulsoup4-4.9.3-py2-none-any.whl | MIT | | big.js | big.js-5.2.2.tgz | MIT | | boolbase | boolbase-1.0.0.tgz | ISC | | twitter-bootstrap | bootstrap-3.3.7.min.js | MIT | | bootstrap | bootstrap-3.4.1.tgz | MIT | | brace-expansion | brace-expansion-1.1.11.tgz | MIT | | braces | braces-2.3.2.tgz | MIT | | bs4 | bs4-0.0.1.tar.gz | MIT | | buffer-from | buffer-from-1.1.1.tgz | MIT | | cache-base | cache-base-1.0.1.tgz | MIT | | cachetools | cachetools-3.1.1-py2.py3-none-any.whl | MIT | | projectcalico | calico-v3.17.1 | Apache 2.0 | | projectcalico | calicoctl-v3.17.1 | Apache 2.0 | | camel-case | camel-case-4.1.1.tgz | MIT | | certifi | certifi-2020.12.5-py2.py3-none-any.whl | Mozilla 2.0 | | chain-function | chain-function-1.0.0.tgz | MIT | | chalk | chalk-2.4.2.tgz | MIT | | chardet | chardet-3.0.4-py2.py3-none-any.whl | LGPL 3.0 | | chardet | chardet-4.0.0-py2.py3-none-any.whl | LGPL 3.0 | | chevrotain | chevrotain-6.5.0.tgz | Apache 2.0 | | cidr-regex | cidr-regex-2.0.10.tgz | BSD 2 | | class-transformer | class-transformer-0.3.1.tgz | MIT | | class-utils | class-utils-0.3.6.tgz | MIT | | class-validator | class-validator-0.9.1.tgz | MIT | | classnames | classnames-2.2.5.tgz | MIT | | classnames | classnames-2.2.6.tgz | MIT | | clean-css | clean-css-4.2.3.tgz | MIT | | clipboard.js | clipboard-2.0.0.min.js | MIT | | cni-plugin | cni-plugin-v3.16.6 | Tigera Proprietary | | @babel/code-frame | code-frame-7.10.1.tgz | MIT | | codemirror | codemirror-5.57.0.js | MIT | | codemirror | codemirror-5.57.0.tgz | MIT | | collection-visit | collection-visit-1.0.0.tgz | MIT | | color-convert | color-convert-1.9.3.tgz | MIT | | color-name | color-name-1.1.3.tgz | MIT | | commander | commander-2.13.0.tgz | MIT | | commander | commander-2.20.3.tgz | MIT | | commander | commander-4.1.1.tgz | MIT | | component-emitter | component-emitter-1.3.0.tgz | MIT | | concat-map | concat-map-0.0.1.tgz | MIT | | configparser | configparser-4.0.2-py2.py3-none-any.whl | MIT | | connected-react-router | connected-react-router-6.5.2.tgz | MIT | | contextlib2 | contextlib2-0.6.0.post1-py2.py3-none-any.whl | Python 2.0 | | copy-descriptor | copy-descriptor-0.1.1.tgz | MIT | | @popperjs/core | core-2.4.4.tgz | MIT | | core-js | core-js-1.2.7.tgz | MIT | | core-js | core-js-2.5.1.tgz | MIT | | core-js | core-js-2.5.7.tgz | MIT | | core-js | core-js-3.6.5.tgz | MIT | | cose-base | cose-base-1.0.3.tgz | MIT | | create-react-class | create-react-class-15.6.2.tgz | MIT | | css-box-model | css-box-model-1.1.1.tgz | MIT | | css-select | css-select-1.2.0.tgz | BSD 2 | | css-what | css-what-2.1.3.tgz | BSD 2 | | @ungap/custom-elements | custom-elements-0.1.12.tgz | ISC | | cytoscape | cytoscape-3.15.2.tgz | MIT | | cytoscape | cytoscape-3.18.0.min.js | MIT | | cytoscape-avsdf | cytoscape-avsdf-1.0.0.tgz | MIT | | cytoscape-cise | cytoscape-cise-1.0.0.tgz | MIT | | cytoscape-context-menus | cytoscape-context-menus-4.0.0.tgz | MIT | | cytoscape-cose-bilkent | cytoscape-cose-bilkent-4.1.0.tgz | MIT | | cytoscape-dagre-cluster-fix | cytoscape-dagre-cluster-fix-2.2.5.tgz | MIT | | cytoscape-expand-collapse | cytoscape-expand-collapse-4.0.0.tgz | MIT | | cytoscape-fcose | cytoscape-fcose-1.2.3.tgz | MIT | | cytoscape-layers | cytoscape-layers-2.1.0.tgz | MIT | | cytoscape-navigator | cytoscape-navigator-2.0.1.tgz | MIT | | cytoscape-popper | cytoscape-popper-1.0.7.js | MIT | | cytoscape-popper | cytoscape-popper-1.0.7.tgz | MIT | | d3 | d3-5.5.0.tgz | BSD 3 | | d3-array | d3-array-1.2.1.tgz | BSD 3 | | d3-array | d3-array-1.2.4.tgz | BSD 3 | | d3-axis | d3-axis-1.0.8.tgz | BSD 3 | | d3-brush | d3-brush-1.0.4.tgz | BSD 3 | | d3-chord | d3-chord-1.0.4.tgz | BSD 3 | | d3-collection | d3-collection-1.0.4.tgz | BSD 3 | | d3-collection | d3-collection-1.0.7.tgz | BSD 3 | | d3-color | d3-color-1.0.3.tgz | BSD 3 | | d3-contour | d3-contour-1.2.0.tgz | BSD 3 | | d3-dispatch | d3-dispatch-1.0.3.tgz | BSD 3 | | d3-drag | d3-drag-1.2.1.tgz | BSD 3 | | d3-dsv | d3-dsv-1.0.8.tgz | BSD 3 | | d3-ease | d3-ease-1.0.3.tgz | BSD 3 | | d3-fetch | d3-fetch-1.1.0.tgz | BSD 3 | | d3-force | d3-force-1.1.0.tgz | BSD 3 | | d3-format | d3-format-1.2.2.tgz | BSD 3 | | d3-geo | d3-geo-1.10.0.tgz | BSD 3 | | d3-hierarchy | d3-hierarchy-1.1.6.tgz | BSD 3 | | d3-interpolate | d3-interpolate-1.1.6.tgz | BSD 3 | | d3-interpolate | d3-interpolate-1.3.2.tgz | BSD 3 | | d3-path | d3-path-1.0.5.tgz | BSD 3 | | d3-polygon | d3-polygon-1.0.3.tgz | BSD 3 | | d3-quadtree | d3-quadtree-1.0.3.tgz | BSD 3 | | d3-random | d3-random-1.1.0.tgz | BSD 3 | | d3-sankey-circular | d3-sankey-circular-0.34.0.tgz | MIT | | d3-scale | d3-scale-2.0.0.tgz | BSD 3 | | d3-scale | d3-scale-2.1.2.tgz | BSD 3 | | d3-scale-chromatic | d3-scale-chromatic-1.3.0.tgz | BSD 3 | | d3-selection | d3-selection-1.3.0.tgz | BSD 3 | | d3-shape | d3-shape-1.2.0.tgz | BSD 3 | | d3-shape | d3-shape-1.2.3.tgz | BSD 3 | | d3-shape | d3-shape-1.3.7.tgz | BSD 3 | | d3-time | d3-time-1.0.8.tgz | BSD 3 | | d3-time-format | d3-time-format-2.1.1.tgz | BSD 3 | | d3-timer | d3-timer-1.0.7.tgz | BSD 3 | | d3-transition | d3-transition-1.1.1.tgz | BSD 3 | | d3-voronoi | d3-voronoi-1.1.2.tgz | BSD 3 | | d3-zoom | d3-zoom-1.7.1.tgz | BSD 3 | | dagre | dagre-0.7.4.js | MIT | | dagre-cluster-fix | dagre-cluster-fix-0.9.3.tgz | MIT | | debug | debug-2.6.9.tgz | MIT | | decimal.js-light | decimal.js-light-2.5.0.tgz | MIT | | decode-uri-component | decode-uri-component-0.2.0.tgz | MIT | | deepdiff | deepdiff-3.3.0-py2-none-any.whl | MIT | | deepmerge | deepmerge-2.1.1.tgz | MIT | | define-properties | define-properties-1.1.2.tgz | MIT | | define-properties | define-properties-1.1.3.tgz | MIT | | define-property | define-property-0.2.5.tgz | MIT | | define-property | define-property-1.0.0.tgz | MIT | | define-property | define-property-2.0.2.tgz | MIT | | diff | diff-3.5.0.tgz | BSD 3 | | diff2html | diff2html-2.4.0.tgz | MIT | | dom-converter | dom-converter-0.2.0.tgz | MIT | | dom-helpers | dom-helpers-3.3.1.tgz | MIT | | dom-helpers | dom-helpers-3.4.0.tgz | MIT | | dom-serializer | dom-serializer-0.2.2.tgz | MIT | | dom-walk | dom-walk-0.1.2.tgz | MIT | | domelementtype | domelementtype-1.3.1.tgz | BSD 2 | | domelementtype | domelementtype-2.0.1.tgz | BSD 2 | | domhandler | domhandler-2.4.2.tgz | BSD 2 | | domutils | domutils-1.5.1.tgz | BSD 2 | | domutils | domutils-1.7.0.tgz | BSD 2 | | dot-case | dot-case-3.0.3.tgz | MIT | | elasticsearch | elasticsearch-6.8.1-py2.py3-none-any.whl | Apache 2.0 | | elementary-circuits-directed-graph | elementary-circuits-directed-graph-1.2.0.tgz | MIT | | emojis-list | emojis-list-3.0.0.tgz | MIT | | encoding | encoding-0.1.12.tgz | MIT | | entities | entities-1.1.2.tgz | BSD 2 | | entities | entities-2.0.3.tgz | BSD 2 | | es-abstract | es-abstract-1.17.5.tgz | MIT | | es-to-primitive | es-to-primitive-1.2.1.tgz | MIT | | es5-shim | es5-shim-4.3.1.js | MIT | | escape-string-regexp | escape-string-regexp-1.0.5.tgz | MIT | | esprima | esprima-4.0.0.tgz | BSD 2 | | expand-brackets | expand-brackets-2.1.4.tgz | MIT | | extend-shallow | extend-shallow-2.0.1.tgz | MIT | | extend-shallow | extend-shallow-3.0.2.tgz | MIT | | extglob | extglob-2.0.4.tgz | MIT | | fast-levenshtein | fast-levenshtein-2.0.6.tgz | MIT | | fbjs | fbjs-0.8.16.tgz | MIT | | felix | felix-v3.17.2 | Tigera Proprietary | | file-saver | file-saver-2.0.1.tgz | MIT | | fill-range | fill-range-4.0.0.tgz | MIT | | @fortawesome/fontawesome-common-types | fontawesome-common-types-0.2.32.tgz | MIT | | @fortawesome/fontawesome-svg-core | fontawesome-svg-core-1.2.10.tgz | MIT | | for-in | for-in-1.0.2.tgz | MIT | | foreach | foreach-2.0.5.tgz | MIT | | fork-ts-checker-webpack-plugin | fork-ts-checker-webpack-plugin-4.1.6.tgz | MIT | | formik | formik-2.1.3.tgz | MIT | | fragment-cache | fragment-cache-0.2.1.tgz | MIT | | @fortawesome/free-brands-svg-icons | free-brands-svg-icons-5.6.1.tgz | CC BY 4.0 | | @fortawesome/free-regular-svg-icons | free-regular-svg-icons-5.15.1.tgz | CC BY 4.0 | | @fortawesome/free-solid-svg-icons | free-solid-svg-icons-5.6.1.tgz | CC BY 4.0 | | funcsigs | funcsigs-1.0.2-py2.py3-none-any.whl | Apache 2.0 | | function-bind | function-bind-1.1.1.tgz | MIT | | get-value | get-value-2.0.6.tgz | MIT | | github.com/alecthomas/participle | github.com/alecthomas/participle-v0.3.0 | MIT | | github.com/apparentlymart/go-cidr/cidr | github.com/apparentlymart/go-cidr/cidr-v1.0.1 | MIT | | github.com/aquasecurity/kube-bench/check | github.com/aquasecurity/kube-bench/check-v0.0.34 | Apache 2.0 | | github.com/araddon/dateparse | github.com/araddon/dateparse-262228af701ebf3932b8b8488da6781b9d585c88 | MIT | | github.com/avast/retry-go | github.com/avast/retry-go-v2.2.0 | MIT | | github.com/aws/aws-lambda-go/events | github.com/aws/aws-lambda-go/events-v1.13.3 | Apache 2.0 | | github.com/aws/aws-sdk-go/aws | github.com/aws/aws-sdk-go/aws-v1.25.8 | Apache 2.0 | | github.com/bmizerany/pat | github.com/bmizerany/pat-6226ea591a40176dd3ff9cd8eff81ed6ca721a00 | MIT | | github.com/bronze1man/gostrongswanvici | github.com/bronze1man/gostrongswanvici-27d02f80ba4008de552efb746b3f6eaa7718b518 | MIT | | github.com/buger/jsonparser | github.com/buger/jsonparser-v1.0.0 | MIT | | github.com/burntsushi/toml | github.com/burntsushi/toml-v0.3.1 | MIT | | github.com/caimeo/iniflags | github.com/caimeo/iniflags-ef4ae6c5cd79d20db0b18bc5ebd8657fac7260e5 | BSD 2 | | github.com/cloudflare/cfssl/log | github.com/cloudflare/cfssl/log-v1.4.1 | BSD 2 | | github.com/containernetworking/cni/libcni | github.com/containernetworking/cni/libcni-v0.8.0 | Apache 2.0 | | github.com/containernetworking/plugins/pkg/hns | github.com/containernetworking/plugins/pkg/hns-v0.8.5 | Apache 2.0 | | github.com/coreos/go-oidc | github.com/coreos/go-oidc-v2.1.0 | Apache 2.0 | | github.com/coreos/go-semver/semver | github.com/coreos/go-semver/semver-v0.3.0 | Apache 2.0 | | github.com/davecgh/go-spew/spew | github.com/davecgh/go-spew/spew-v1.1.1 | ISC | | github.com/docker/docker/api/types | github.com/docker/docker/api/types-v1.13.1 | Apache 2.0 | | github.com/docker/docker/client | github.com/docker/docker/client-v1.13.1 | Apache 2.0 | | github.com/docker/go-connections/nat | github.com/docker/go-connections/nat-v0.4.0 | Apache 2.0 | | github.com/docopt/docopt-go | github.com/docopt/docopt-go-ee0de3bc6815ee19d4a46c7eb90f829db0e014b1 | MIT | | github.com/elastic/go-elasticsearch/v7 | github.com/elastic/go-elasticsearch/v7-v7.3.0 | Apache 2.0 | | github.com/envoyproxy/data-plane-api/envoy/api/v2/core | github.com/envoyproxy/data-plane-api/envoy/api/v2/core-ffd420ef8a9ad148642236aa6d89e2855b41c821 | Apache 2.0 | | github.com/fsnotify/fsnotify | github.com/fsnotify/fsnotify-v1.4.9 | BSD 3 | | github.com/gavv/monotime | github.com/gavv/monotime-30dba43534243e3484a34676a0f068d12b989f84 | Apache 2.0 | | github.com/getlantern/deepcopy | github.com/getlantern/deepcopy-v1 | Apache 2.0 | | github.com/ghodss/yaml | github.com/ghodss/yaml-v1.0.0 | BSD 3 | | github.com/go-ini/ini | github.com/go-ini/ini-v1.43.0 | Apache 2.0 | | github.com/go-logr/logr | github.com/go-logr/logr-v0.3.0 | Apache 2.0 | | github.com/go-openapi/spec | github.com/go-openapi/spec-v0.19.3 | Apache 2.0 | | github.com/go-sql-driver/mysql | github.com/go-sql-driver/mysql-v1.4.1 | Mozilla 2.0 | | github.com/gofrs/flock | github.com/gofrs/flock-v0.8.0 | BSD 3 | | github.com/gogo/googleapis/google/rpc | github.com/gogo/googleapis/google/rpc-v1.2.0 | Apache 2.0 | | github.com/gogo/protobuf/proto | github.com/gogo/protobuf/proto-v1.3.1 | BSD 3 | | github.com/golang-collections/collections/stack | github.com/golang-collections/collections/stack-604e922904d35e97f98a774db7881f049cd8d970 | MIT | | github.com/google/go-cmp/cmp | github.com/google/go-cmp/cmp-v0.4.0 | BSD 3 | | github.com/google/gofuzz | github.com/google/gofuzz-v1.1.0 | Apache 2.0 | | github.com/google/gopacket | github.com/google/gopacket-v1.1.18 | BSD 3 | | github.com/google/netstack/tcpip/header | github.com/google/netstack/tcpip/header-55fcc16cd0eb096d8418f7bc5162483c31a4e82b | Apache 2.0 | | github.com/hashicorp/go-version | github.com/hashicorp/go-version-v1.2.1 | Mozilla 2.0 | | github.com/hashicorp/golang-lru | github.com/hashicorp/golang-lru-v0.5.1 | Mozilla 2.0 | | github.com/hashicorp/yamux | github.com/hashicorp/yamux-2f1d1f20f75d5404f53b9edf6b53ed5505508675 | Mozilla 2.0 | | github.com/howeyc/fsnotify | github.com/howeyc/fsnotify-v0.9.0 | BSD 3 | | github.com/hpcloud/tail | github.com/hpcloud/tail-v1.0.0 | MIT | | github.com/ishidawataru/sctp | github.com/ishidawataru/sctp-00ab2ac2db07a138417639ef3f39672c65dbb9a0 | BSD 3 | | github.com/jarcoal/httpmock | github.com/jarcoal/httpmock-v1.0.5 | MIT | | github.com/jinzhu/copier | github.com/jinzhu/copier-v0.1.0 | MIT | | github.com/jmespath/go-jmespath | github.com/jmespath/go-jmespath-0.3.0 | Apache 2.0 | | github.com/joho/godotenv | github.com/joho/godotenv-v1.3.0 | MIT | | github.com/jpillora/backoff | github.com/jpillora/backoff-v1.0.0 | MIT | | github.com/json-iterator/go | github.com/json-iterator/go-v1.1.10 | MIT | | github.com/juju/clock | github.com/juju/clock-9c5c9712527c7986f012361e7d13756b4d99543d | LGPL 3.0 | | github.com/juju/errors | github.com/juju/errors-3fe23663418fc1d724868c84f21b7519bbac7441 | LGPL 3.0 | | github.com/juju/mutex | github.com/juju/mutex-d21b13acf4bfd8a8b0482a3a78e44d98880b40d3 | LGPL 3.0 | | github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/utils | github.com/k8snetworkplumbingwg/network-attachment-definition-client/pkg/utils-v1.1.0 | Apache 2.0 | | github.com/kardianos/osext | github.com/kardianos/osext-2bc1f35cddc0cc527b4bc3dce8578fc2a6c11384 | BSD 3 | | github.com/kelseyhightower/envconfig | github.com/kelseyhightower/envconfig-v1.4.0 | MIT | | github.com/kelseyhightower/memkv | github.com/kelseyhightower/memkv-v0.1.1 | MIT | | github.com/konsorten/go-windows-terminal-sequences | github.com/konsorten/go-windows-terminal-sequences-v1.0.1 | MIT | | github.com/lestrrat-go/file-rotatelogs | github.com/lestrrat-go/file-rotatelogs-v2.4.0 | MIT | | github.com/libp2p/go-reuseport | github.com/libp2p/go-reuseport-v0.0.1 | ISC | | github.com/lithammer/dedent | github.com/lithammer/dedent-v1.1.0 | MIT | | github.com/mailru/easyjson | github.com/mailru/easyjson-v0.7.0 | MIT | | github.com/masterminds/sprig | github.com/masterminds/sprig-v2.19.0 | MIT | | github.com/mcuadros/go-version | github.com/mcuadros/go-version-92cdf37c5b7579ebaf7a036da94b40995972088d | MIT | | github.com/microsoft/hcsshim | github.com/microsoft/hcsshim-v0.8.6 | MIT | | github.com/mipearson/rfw | github.com/mipearson/rfw-6f0a6f3266ba1058df9ef0c94cda1cecd2e62852 | MIT | | github.com/mitchellh/go-homedir | github.com/mitchellh/go-homedir-v1.1.0 | MIT | | github.com/modern-go/concurrent | github.com/modern-go/concurrent-1.0.3 | Apache 2.0 | | github.com/modern-go/reflect2 | github.com/modern-go/reflect2-v1.0.1 | Apache 2.0 | | github.com/natefinch/atomic | github.com/natefinch/atomic-a62ce929ffcc871a51e98c6eba7b20321e3ed62d | MIT | | github.com/nmrshll/go-cp | github.com/nmrshll/go-cp-61436d3b7cfa1bc1e8e455c35d8f60b8e51ccc2e | MIT | | github.com/nxadm/tail | github.com/nxadm/tail-v1.4.4 | MIT | | github.com/olekukonko/tablewriter | github.com/olekukonko/tablewriter-v0.0.2 | MIT | | github.com/olivere/elastic/v7 | github.com/olivere/elastic/v7-v7.0.6 | MIT | | github.com/onsi/ginkgo | github.com/onsi/ginkgo-v1.15.0 | MIT | | github.com/onsi/gomega | github.com/onsi/gomega-v1.7.0 | MIT | | github.com/openshift/api/config/v1 | github.com/openshift/api/config/v1-d0822898eabb929c40c5146116252477abab8d18 | Apache 2.0 | | github.com/openshift/library-go/pkg/crypto | github.com/openshift/library-go/pkg/crypto-9350cd67a9110bcaf9a85d391fa264afbbff1342 | Apache 2.0 | | github.com/osrg/gobgp/client | github.com/osrg/gobgp/client-v1.22 | Apache 2.0 | | github.com/paloaltonetworks/pango | github.com/paloaltonetworks/pango-v0.1.1 | ISC | | github.com/patrickmn/go-cache | github.com/patrickmn/go-cache-v2.1.0 | MIT | | github.com/pkg/errors | github.com/pkg/errors-v0.8.1 | BSD 2 | | github.com/projectcalico/cni-plugin | github.com/projectcalico/cni-plugin | Apache 2.0 | | github.com/projectcalico/felix | github.com/projectcalico/felix | Apache 2.0 | | github.com/projectcalico/go-json | github.com/projectcalico/go-json/json-6219dc7339ba20ee4c57df0a8baac62317d19cb1 | BSD 2 | | github.com/projectcalico/go-yaml-wrapper | github.com/projectcalico/go-yaml-wrapper-090425220c545f6d179db17af395f5aac30b6926 | BSD 3 | | kube-controllers | kube-controllers | Tigera Proprietary | | github.com/projectcalico/libcalico-go | github.com/projectcalico/libcalico-go/lib | Apache 2.0 | | github.com/projectcalico/pod2daemon | github.com/projectcalico/pod2daemon | Apache 2.0 | | typha | typha | Tigera Proprietary | | github.com/prometheus/client\_golang/ | github.com/prometheus/client\_golang/prometheus-v1.7.1 | Apache 2.0 | | github.com/rakelkar/gonetsh/netsh | github.com/rakelkar/gonetsh/netsh-e5c5ffe4bdf04bc060fc45ff4aca2349f51c94a7 | Apache 2.0 | | github.com/robfig/cron | github.com/robfig/cron-v1.2.0 | MIT | | github.com/satori/go.uuid | github.com/satori/go.uuid-v1.2.0 | MIT | | github.com/shirou/gopsutil/process | github.com/shirou/gopsutil/process-v2.19.03 | BSD 3 | | github.com/sirupsen/logrus | github.com/sirupsen/logrus-v1.4.2 | MIT | | github.com/sirupsen/logrus | github.com/sirupsen/logrus-v1.6.0 | MIT | | github.com/spf13/cobra | github.com/spf13/cobra-v0.0.3 | Apache 2.0 | | github.com/spf13/cobra | github.com/spf13/cobra-v1.0.0 | Apache 2.0 | | github.com/spf13/pflag | github.com/spf13/pflag-v1.0.5 | BSD 3 | | github.com/spf13/viper | github.com/spf13/viper-v1.6.1 | MIT | | github.com/stretchr/testify | github.com/stretchr/testify/mock-v1.4.0 | MIT | | github.com/termie/go-shutil | github.com/termie/go-shutil-bcacb06fecaeec8dc42af03c87c6949f4a05c74c | MIT | | github.com/tigera/api | github.com/tigera/api/ | Apache 2.0 | | github.com/vishvananda/netlink | github.com/vishvananda/netlink-v1.1.0 | Apache 2.0 | | github.com/willf/bitset | github.com/willf/bitset-v1.1.11 | BSD 3 | | github.com/workiva/go-datastructures/trie/ctrie | github.com/workiva/go-datastructures/trie/ctrie-v1.0.50 | Apache 2.0 | | github.com/x-cray/logrus-prefixed-formatter | github.com/x-cray/logrus-prefixed-formatter-v0.5.2 | MIT | | github.com/yalp/jsonpath | github.com/yalp/jsonpath-5cc68e5049a040829faef3a44c00ec4332f6dec7 | BSD 3 | | global | global-4.4.0.tgz | MIT | | go.etcd.io/etcd/ | go.etcd.io/etcd/client-v0.5.0-alpha.5.0.20201125193152-8a03d2e9614b | Apache 2.0 | | go.uber.org/zap | go.uber.org/zap-v1.15.0 | MIT | | golang.org/x/crypto | golang.org/x/crypto/ | BSD 3 | | golang.org/x/net | golang.org/x/net/ | BSD 3 | | golang.org/x/sync/ | golang.org/x/sync/ | BSD 3 | | golang.org/x/sys | golang.org/x/sys | BSD 3 | | github.com/golang/text | golang.org/x/text | Golang BSD + Patents | | golang.zx2c4.com/wireguard/ | golang.zx2c4.com/wireguard/ | MIT | | google-libphonenumber | google-libphonenumber-3.2.2.tgz | Apache 2.0 | | google.golang.org/grpc | google.golang.org/grpc-v1.27.0 | Apache 2.0 | | google-auth | google\_auth-1.26.1-py2.py3-none-any.whl | Apache 2.0 | | gopkg.in/fsnotify/fsnotify.v1 | gopkg.in/fsnotify/fsnotify.v1-v1.4.7 | BSD 3 | | gopkg.in/go-playground/validator.v9 | gopkg.in/go-playground/validator.v9-v9.30.2 | MIT | | gopkg.in/inf.v0 | gopkg.in/inf.v0-v0.9.0 | BSD 3 | | gopkg.in/natefinch/lumberjack.v2 | gopkg.in/natefinch/lumberjack.v2-v2.0.0 | MIT | | gopkg.in/square/go-jose.v2 | gopkg.in/square/go-jose.v2-v2.2.3 | Apache 2.0 | | gopkg.in/tchap/go-patricia.v2/patricia | gopkg.in/tchap/go-patricia.v2/patricia-v2.3.0 | MIT | | gopkg.in/tomb.v1 | gopkg.in/tomb.v1-dd632973f1e7218eb1089048e0798ec9ae7dceb8 | BSD 3 | | gopkg.in/yaml.v2 | gopkg.in/yaml.v2-v2.4.0 | Apache 2.0 | | graphlib | graphlib-2.1.8.tgz | MIT | | gud | gud-1.0.0.tgz | MIT | | has | has-1.0.3.tgz | MIT | | has-flag | has-flag-3.0.0.tgz | MIT | | has-symbols | has-symbols-1.0.1.tgz | MIT | | has-value | has-value-0.3.1.tgz | MIT | | has-value | has-value-1.0.0.tgz | MIT | | has-values | has-values-0.1.4.tgz | MIT | | has-values | has-values-1.0.0.tgz | MIT | | he | he-1.2.0.tgz | MIT | | heap | heap-0.2.6.tgz | Python 2.0 | | @babel/helper-validator-identifier | helper-validator-identifier-7.10.1.tgz | MIT | | @babel/highlight | highlight-7.10.1.tgz | MIT | | history | history-4.9.0.tgz | MIT | | twitter | hogan.js | Apache 2.0 | | hogan.js | hogan.js-3.0.2.tgz | Apache 2.0 | | hoist-non-react-statics | hoist-non-react-statics-3.1.0.tgz | BSD 3 | | hoist-non-react-statics | hoist-non-react-statics-3.3.0.tgz | BSD 3 | | @types/html-minifier-terser | html-minifier-terser-5.1.0.tgz | MIT | | html-minifier-terser | html-minifier-terser-5.1.1.tgz | MIT | | html-webpack-plugin | html-webpack-plugin-4.3.0.tgz | MIT | | htmlparser2 | htmlparser2-3.10.1.tgz | MIT | | humps | humps-2.0.1.tgz | MIT | | icepick | icepick-1.3.0.tgz | MIT | | iconv-lite | iconv-lite-0.4.23.tgz | MIT | | idna | idna-2.10-py2.py3-none-any.whl | BSD 3 | | idna | idna-2.7-py2.py3-none-any.whl | BSD 2 | | immutable | immutable-3.8.2.tgz | MIT | | importlib-metadata | importlib\_metadata-2.1.1-py2.py3-none-any.whl | Apache 2.0 | | inherits | inherits-2.0.4.tgz | ISC | | instantsearch.js | instantsearch.production-4.4.1.min.js | MIT | | invariant | invariant-2.2.2.tgz | BSD 3 | | invariant | invariant-2.2.4.tgz | MIT | | ip-regex | ip-regex-2.1.0.tgz | MIT | | ip-regex | ip-regex-4.1.0.tgz | MIT | | ipaddr.js | ipaddr.js-1.9.1.tgz | MIT | | ipaddress | ipaddress-1.0.23-py2.py3-none-any.whl | Python 2.0 | | is-accessor-descriptor | is-accessor-descriptor-0.1.6.tgz | MIT | | is-accessor-descriptor | is-accessor-descriptor-1.0.0.tgz | MIT | | is-buffer | is-buffer-1.1.6.tgz | MIT | | is-callable | is-callable-1.1.5.tgz | MIT | | is-cidr | is-cidr-3.1.0.tgz | BSD 2 | | is-data-descriptor | is-data-descriptor-0.1.4.tgz | MIT | | is-data-descriptor | is-data-descriptor-1.0.0.tgz | MIT | | is-date-object | is-date-object-1.0.1.tgz | MIT | | is-descriptor | is-descriptor-0.1.6.tgz | MIT | | is-descriptor | is-descriptor-1.0.2.tgz | MIT | | is-extendable | is-extendable-0.1.1.tgz | MIT | | is-extendable | is-extendable-1.0.1.tgz | MIT | | is-ip | is-ip-3.1.0.tgz | MIT | | is-number | is-number-3.0.0.tgz | MIT | | is-plain-object | is-plain-object-2.0.4.tgz | MIT | | is-regex | is-regex-1.0.5.tgz | MIT | | is-stream | is-stream-1.1.0.tgz | MIT | | is-symbol | is-symbol-1.0.3.tgz | MIT | | is-windows | is-windows-1.0.2.tgz | MIT | | isarray | isarray-0.0.1.tgz | MIT | | isarray | isarray-1.0.0.tgz | MIT | | isobject | isobject-2.1.0.tgz | MIT | | isobject | isobject-3.0.1.tgz | MIT | | isomorphic-fetch | isomorphic-fetch-2.2.1.tgz | MIT | | jquery | jquery-2.2.0.min.js | MIT | | jquery | jquery-3.4.0.min.js | MIT | | js-cookie | js-cookie-2.2.1.tgz | MIT | | js-tokens | js-tokens-3.0.2.tgz | MIT | | js-tokens | js-tokens-4.0.0.tgz | MIT | | js-yaml | js-yaml-3.14.0.tgz | MIT | | jsan | jsan-3.1.13.tgz | MIT | | json5 | json5-1.0.1.tgz | MIT | | jsonpickle | jsonpickle-2.0.0-py2.py3-none-any.whl | BSD 2 | | jsrsasign | jsrsasign-5.1.0.tgz | MIT | | k8s.io/api | k8s.io/api | Apache 2.0 | | k8s.io/apiextensions-apiserver/ | k8s.io/apiextensions-apiserver/ | Apache 2.0 | | k8s.io/apimachinery | k8s.io/apimachinery/ | Apache 2.0 | | k8s.io/apiserver | k8s.io/apiserver | Apache 2.0 | | k8s.io/client-go | k8s.io/client-go/ | Apache 2.0 | | k8s.io/component-base/ | k8s.io/component-base/ | Apache 2.0 | | k8s.io/klog | k8s.io/klog-v1.0.0 | Apache 2.0 | | k8s.io/kube-aggregator/ | k8s.io/kube-aggregator/ | Apache 2.0 | | k8s.io/kube-openapi/ | k8s.io/kube-openapi | Apache 2.0 | | k8s.io/kubernetes/ | k8s.io/kubernetes/ | Apache 2.0 | | k8s.io/utils/strings | k8s.io/utils/strings | Apache 2.0 | | keycode | keycode-2.1.9.tgz | MIT | | kind-of | kind-of-3.2.2.tgz | MIT | | kind-of | kind-of-4.0.0.tgz | MIT | | kind-of | kind-of-5.1.0.tgz | MIT | | kind-of | kind-of-6.0.3.tgz | MIT | | kube-controllers | kube-controllers-v3.0.11 | Tigera Proprietary | | kube-controllers | kube-controllers-v3.16.6 | Tigera Proprietary | | kubernetes | kubernetes-12.0.1-py2.py3-none-any.whl | Apache 2.0 | | layout-base | layout-base-1.0.2.tgz | MIT | | projectcalico | libcalico-go-v3.18.0-0.dev | Apache 2.0 | | loader-utils | loader-utils-1.4.0.tgz | MIT | | lodash | lodash-4.17.19.tgz | MIT | | lodash | lodash-4.17.20.tgz | MIT | | lodash-es | lodash-es-4.17.15.tgz | MIT | | lodash.debounce | lodash.debounce-4.0.8.tgz | MIT | | lodash.get | lodash.get-4.4.2.tgz | MIT | | lodash.isequal | lodash.isequal-4.5.0.tgz | MIT | | lodash.throttle | lodash.throttle-4.1.1.tgz | MIT | | lodash.topath | lodash.topath-4.5.2.tgz | MIT | | pimterry | loglevel-v1.6.8 | MIT | | loose-envify | loose-envify-1.3.1.tgz | MIT | | loose-envify | loose-envify-1.4.0.tgz | MIT | | lower-case | lower-case-2.0.1.tgz | MIT | | map-cache | map-cache-0.2.2.tgz | MIT | | map-visit | map-visit-1.0.0.tgz | MIT | | math-expression-evaluator | math-expression-evaluator-1.2.17.tgz | MIT | | megacubo | megacubo-br-Megacubo\_15.4.7\_linux\_ia32 | LGPL 2.1 | | memoize-one | memoize-one-4.0.3.tgz | MIT | | microevent.ts | microevent.ts-0.1.1.tgz | MIT | | micromatch | micromatch-3.1.10.tgz | MIT | | min-document | min-document-2.19.0.tgz | MIT | | mini-create-react-context | mini-create-react-context-0.3.2.tgz | MIT | | minimatch | minimatch-3.0.4.tgz | ISC | | minimist | minimist-1.2.5.tgz | MIT | | mixin-deep | mixin-deep-1.3.2.tgz | MIT | | mkdirp | mkdirp-0.3.0.tgz | MIT X11 | | mocha | mocha-1.6.0.js | MIT | | moment | moment-2.22.2.tgz | MIT | | more-itertools | more\_itertools-5.0.0-py2-none-any.whl | MIT | | ms | ms-2.0.0.tgz | MIT | | nanoid | nanoid-2.1.7.tgz | MIT | | nanomatch | nanomatch-1.2.13.tgz | MIT | | netaddr | netaddr-0.7.19-py2.py3-none-any.whl | BSD 3 | | no-case | no-case-3.0.3.tgz | MIT | | @types/node | node-9.3.0.tgz | MIT | | node-fetch | node-fetch-1.7.3.tgz | MIT | | nodejs | node-v10.23.1 | Node.js | | node | node-v3.17.2 | Tigera Proprietary | | nopt | nopt-1.0.10.tgz | MIT | | nose-timer | nose-timer-0.7.1.tar.gz | MIT | | nose-parameterized | nose\_parameterized-0.6.0-py2.py3-none-any.whl | BSD 3 | | nth-check | nth-check-1.0.2.tgz | BSD 2 | | oauthlib | oauthlib-3.1.0-py2.py3-none-any.whl | BSD 3 | | object-assign | object-assign-4.1.1.tgz | MIT | | object-copy | object-copy-0.1.0.tgz | MIT | | object-inspect | object-inspect-1.7.0.tgz | MIT | | object-keys | object-keys-1.0.11.tgz | MIT | | object-keys | object-keys-1.1.1.tgz | MIT | | object-visit | object-visit-1.0.1.tgz | MIT | | object.assign | object.assign-4.1.0.tgz | MIT | | object.getownpropertydescriptors | object.getownpropertydescriptors-2.1.0.tgz | MIT | | object.pick | object.pick-1.3.0.tgz | MIT | | oidc-client | oidc-client-1.4.1.tgz | Apache 2.0 | | openshift | origin-v3.6.1 | Apache 2.0 | | packaging | packaging-20.9-py2.py3-none-any.whl | BSD 2 | | param-case | param-case-3.0.3.tgz | MIT | | parse-duration | parse-duration-0.4.4.tgz | MIT | | pascal-case | pascal-case-3.1.1.tgz | MIT | | pascalcase | pascalcase-0.1.1.tgz | MIT | | path-to-regexp | path-to-regexp-1.7.0.tgz | MIT | | pathlib2 | pathlib2-2.3.5-py2.py3-none-any.whl | MIT | | pegjs | pegjs-0.10.0.tgz | MIT | | performance-now | performance-now-2.1.0.tgz | MIT | | pluggy | pluggy-0.13.1-py2.py3-none-any.whl | MIT | | popper.js | popper-1.16.0.js | MIT | | popper.js | popper.js-1.16.1.tgz | MIT | | posix-character-classes | posix-character-classes-0.1.1.tgz | MIT | | pretty-error | pretty-error-2.1.1.tgz | MIT | | process | process-0.11.10.tgz | MIT | | promise | promise-7.3.1.tgz | MIT | | prop-types | prop-types-15.5.10.js | BSD 3 | | prop-types | prop-types-15.6.0.tgz | MIT | | prop-types | prop-types-15.6.2.js | BSD 3 | | prop-types | prop-types-15.6.2.tgz | MIT | | prop-types | prop-types-15.7.2.tgz | MIT | | prop-types-extra | prop-types-extra-1.0.1.tgz | MIT | | py | py-1.10.0-py2.py3-none-any.whl | MIT | | pyasn1 | pyasn1-0.4.8-py2.py3-none-any.whl | BSD 2 | | pyasn1-modules | pyasn1\_modules-0.2.8-py2.py3-none-any.whl | BSD 2 | | pyparsing | pyparsing-2.4.7-py2.py3-none-any.whl | MIT | | pytest | pytest-4.6.11-py2.py3-none-any.whl | MIT | | python-dateutil | python\_dateutil-2.8.1-py2.py3-none-any.whl | BSD 3 | | PyYAML | PyYAML-5.4.1-cp27-cp27mu-manylinux1\_x86\_64.whl | MIT | | raf | raf-3.4.0.tgz | MIT | | raf-schd | raf-schd-4.0.0.tgz | MIT | | raven-js | raven-js-3.22.1.tgz | BSD 2 | | react | react-15.6.1.js | MIT | | react | react-16.13.1.tgz | MIT | | react | react-16.8.6.tgz | MIT | | react-beautiful-dnd | react-beautiful-dnd-10.0.2.tgz | Apache 2.0 | | react-bootstrap | react-bootstrap-0.32.1.tgz | MIT | | react-codemirror2 | react-codemirror2-5.1.0.tgz | MIT | | react-confirm | react-confirm-0.1.16.tgz | MIT | | react-cytoscapejs | react-cytoscapejs-1.2.1.tgz | MIT | | plotly | react-cytoscapejs-v1.2.1 | MIT | | react-day-picker | react-day-picker-7.4.8.tgz | MIT | | react-dom | react-dom-15.6.1.js | MIT | | react-dom | react-dom-16.13.1.tgz | MIT | | react-dom | react-dom-16.8.6.tgz | MIT | | react-draggable | react-draggable-3.0.5.tgz | MIT | | react-fast-compare | react-fast-compare-2.0.4.tgz | MIT | | react-filter-box | react-filter-box-3.4.1.tgz | MIT | | @fortawesome/react-fontawesome | react-fontawesome-0.1.3.tgz | MIT | | react-grid-layout | react-grid-layout-0.16.3.tgz | MIT | | react-hot-loader | react-hot-loader-4.12.21.tgz | MIT | | react-input-autosize | react-input-autosize-2.1.2.tgz | MIT | | JedWatson | react-input-autosize-v2.1.2 | MIT | | react-is | react-is-16.6.3.tgz | MIT | | react-is | react-is-16.8.6.tgz | MIT | | react-is | react-is-16.9.0.tgz | MIT | | react-json-pretty | react-json-pretty-2.2.0.tgz | MIT | | react-lifecycles-compat | react-lifecycles-compat-3.0.4.tgz | MIT | | react-native-segmented-control-tab | react-native-segmented-control-tab-3.2.1.tgz | MIT | | react-new-window | react-new-window-0.1.2.tgz | MIT | | react-notification-system | react-notification-system-0.2.17.tgz | MIT | | react-overlays | react-overlays-0.8.3.tgz | MIT | | react-prop-types | react-prop-types-0.4.0.tgz | MIT | | react-querybuilder | react-querybuilder-3.0.0.tgz | MIT | | react-redux | react-redux-5.1.1.tgz | MIT | | react-redux | react-redux-7.1.1.tgz | MIT | | react-redux-form | react-redux-form-1.16.5.tgz | MIT | | react-resizable | react-resizable-1.7.5.tgz | MIT | | react-resize-detector | react-resize-detector-2.3.0.tgz | MIT | | react-router | react-router-5.0.1.tgz | MIT | | react-router-dom | react-router-dom-5.0.1.tgz | MIT | | react-show | react-show-2.0.4.tgz | MIT | | react-smooth | react-smooth-1.0.2.tgz | MIT | | react-split-pane | react-split-pane-0.1.92.tgz | MIT | | react-style-proptype | react-style-proptype-3.2.2.tgz | MIT | | react-switch | react-switch-5.0.0.tgz | MIT | | react-table-6 | react-table-6-6.11.0.tgz | MIT | | react-table | react-table-7.5.1.tgz | MIT | | react-tooltip | react-tooltip-4.2.11.tgz | MIT | | react-transition-group | react-transition-group-2.2.1.tgz | BSD 3 | | react-transition-group | react-transition-group-2.7.1.tgz | BSD 3 | | readable-stream | readable-stream-3.6.0.tgz | MIT | | recharts | recharts-1.5.0.tgz | MIT | | recharts-scale | recharts-scale-0.4.2.tgz | MIT | | reduce-css-calc | reduce-css-calc-1.3.0.tgz | MIT | | reduce-function-call | reduce-function-call-1.0.2.tgz | MIT | | redux | redux-4.0.1.tgz | MIT | | redux | redux-4.0.4.tgz | MIT | | redux-immutable | redux-immutable-4.0.0.tgz | BSD 3 | | redux-merge-immutable-reducers | redux-merge-immutable-reducers-0.1.4.tgz | MIT | | redux-thunk | redux-thunk-2.3.0.tgz | MIT | | reflect-metadata | reflect-metadata-0.1.13.tgz | Apache 2.0 | | regenerator-runtime | regenerator-runtime-0.11.1.tgz | MIT | | regenerator-runtime | regenerator-runtime-0.12.1.tgz | MIT | | regenerator-runtime | regenerator-runtime-0.13.3.tgz | MIT | | regenerator-runtime | regenerator-runtime-0.13.5.tgz | MIT | | regex-not | regex-not-1.0.2.tgz | MIT | | regexp-to-ast | regexp-to-ast-0.4.0.tgz | MIT | | relateurl | relateurl-0.2.7.tgz | MIT | | remotedev-serialize | remotedev-serialize-0.1.8.tgz | MIT | | renderkid | renderkid-2.0.3.tgz | MIT | | repeat-element | repeat-element-1.1.3.tgz | MIT | | repeat-string | repeat-string-1.6.1.tgz | MIT | | requests | requests-2.20.1-py2.py3-none-any.whl | Apache 2.0 | | requests | requests-2.25.1-py2.py3-none-any.whl | Apache 2.0 | | requests-oauthlib | requests\_oauthlib-1.3.0-py2.py3-none-any.whl | ISC | | reselect | reselect-2.5.4.tgz | MIT | | @juggle/resize-observer | resize-observer-3.2.0.tgz | Apache 2.0 | | resize-observer-polyfill | resize-observer-polyfill-1.5.1.tgz | MIT | | resolve-pathname | resolve-pathname-2.2.0.tgz | MIT | | resolve-url | resolve-url-0.2.1.tgz | MIT | | ret | ret-0.1.15.tgz | MIT | | rsa | rsa-4.5-py2.py3-none-any.whl | Apache 2.0 | | @babel/runtime | runtime-7.1.5.tgz | MIT | | @babel/runtime | runtime-7.5.5.tgz | MIT | | @babel/runtime-corejs2 | runtime-corejs2-7.1.5.tgz | MIT | | rw | rw-1.3.3.tgz | BSD 3 | | safe-buffer | safe-buffer-5.2.1.tgz | MIT | | safe-regex | safe-regex-1.1.0.tgz | MIT | | safer-buffer | safer-buffer-2.1.2.tgz | MIT | | scandir | scandir-1.10.0.tar.gz | BSD 3 | | scheduler | scheduler-0.13.6.tgz | MIT | | scheduler | scheduler-0.18.0.tgz | MIT | | scheduler | scheduler-0.19.1.tgz | MIT | | seamless-immutable | seamless-immutable-7.1.4.tgz | BSD 3 | | semver | semver-5.7.1.tgz | ISC | | set-value | set-value-2.0.1.tgz | MIT | | setimmediate | setimmediate-1.0.5.tgz | MIT | | setuptools | setuptools-44.1.1-py2.py3-none-any.whl | MIT | | shallow-compare | shallow-compare-1.2.2.tgz | MIT | | shallowequal | shallowequal-1.1.0.tgz | MIT | | sigs.k8s.io/controller-runtime | sigs.k8s.io/controller-runtime-v0.7.0 | Apache 2.0 | | sigs.k8s.io/kind/pkg/errors | sigs.k8s.io/kind/pkg/errors-v0.9.0 | Apache 2.0 | | sigs.k8s.io/yaml | sigs.k8s.io/yaml-v1.2.0 | BSD 3 | | simplejson | simplejson-3.13.2.tar.gz | Academic 2.1 | | six | six-1.15.0-py2.py3-none-any.whl | MIT | | snapdragon | snapdragon-0.8.2.tgz | MIT | | snapdragon-node | snapdragon-node-2.1.1.tgz | MIT | | snapdragon-util | snapdragon-util-3.0.1.tgz | MIT | | soupsieve | soupsieve-1.9.6-py2.py3-none-any.whl | MIT | | @types/source-list-map | source-list-map-0.1.2.tgz | MIT | | source-map | source-map-0.5.7.tgz | BSD 3 | | source-map | source-map-0.6.1.tgz | BSD 3 | | source-map | source-map-0.7.3.tgz | BSD 3 | | source-map-resolve | source-map-resolve-0.5.3.tgz | MIT | | source-map-support | source-map-support-0.5.19.tgz | MIT | | source-map-url | source-map-url-0.4.0.tgz | MIT | | split-string | split-string-3.1.0.tgz | MIT | | sprintf | sprintf-1.0.3.js | BSD | | sprintf-js | sprintf-js-1.0.3.tgz | BSD 3 | | static-extend | static-extend-0.1.2.tgz | MIT | | string.prototype.trimend | string.prototype.trimend-1.0.1.tgz | MIT | | string.prototype.trimleft | string.prototype.trimleft-2.1.2.tgz | MIT | | string.prototype.trimright | string.prototype.trimright-2.1.2.tgz | MIT | | string.prototype.trimstart | string.prototype.trimstart-1.0.1.tgz | MIT | | string\_decoder | string\_decoder-1.3.0.tgz | MIT | | strip-ansi | strip-ansi-3.0.1.tgz | MIT | | strongly-connected-components | strongly-connected-components-1.0.1.tgz | MIT | | supports-color | supports-color-5.5.0.tgz | MIT | | swagger-ui | swagger-ui-bundle-3.37.0.js | Apache 2.0 | | swagger-ui | swagger-ui-standalone-preset-3.37.0.js | Apache 2.0 | | symbol-observable | symbol-observable-1.2.0.tgz | MIT | | @types/tapable | tapable-1.0.5.tgz | MIT | | tapable | tapable-1.1.3.tgz | MIT | | Jstarfish | Technical-Learning-609d9d75ca68e30aee8757b26f52bf132c644be7 | ISC | | termcolor | termcolor-1.1.0.tar.gz | MIT | | terser | terser-4.7.0.tgz | BSD 2 | | tiny-invariant | tiny-invariant-1.0.3.tgz | MIT | | tiny-invariant | tiny-invariant-1.0.6.tgz | MIT | | tiny-warning | tiny-warning-1.0.2.tgz | MIT | | tiny-warning | tiny-warning-1.0.3.tgz | MIT | | tippy.js | tippy-bundle.iife-5.2.1.min.js | MIT | | tippy.js | tippy.js-6.2.5.tgz | MIT | | to-object-path | to-object-path-0.3.0.tgz | MIT | | to-regex | to-regex-3.0.2.tgz | MIT | | to-regex-range | to-regex-range-2.1.1.tgz | MIT | | tslib | tslib-1.10.0.js | Apache 2.0 | | tslib | tslib-1.10.0.tgz | Apache 2.0 | | tslib | tslib-1.13.0.js | Apache 2.0 | | tslib | tslib.es6-1.10.0.js | Apache 2.0 | | tslib | tslib.es6-1.13.0.js | Apache 2.0 | | typescript-fsa | typescript-fsa-2.5.0.tgz | MIT | | typescript-fsa-reducers | typescript-fsa-reducers-0.4.5.tgz | MIT | | ua-parser-js | ua-parser-js-0.7.18.tgz | MIT | | @types/uglify-js | uglify-js-3.9.2.tgz | MIT | | uncontrollable | uncontrollable-4.1.0.tgz | MIT | | union-value | union-value-1.0.1.tgz | MIT | | unset-value | unset-value-1.0.0.tgz | MIT | | urix | urix-0.1.0.tgz | MIT | | urllib3 | urllib3-1.24.3-py2.py3-none-any.whl | MIT | | urllib3 | urllib3-1.26.3-py2.py3-none-any.whl | MIT | | use | use-3.1.1.tgz | MIT | | util-deprecate | util-deprecate-1.0.2.tgz | MIT | | util.promisify | util.promisify-1.0.0.tgz | MIT | | utila | utila-0.4.0.tgz | MIT | | uuid | uuid-7.0.3.tgz | MIT | | validator | validator-10.4.0.tgz | MIT | | value-equal | value-equal-0.4.0.tgz | MIT | | warning | warning-3.0.0.tgz | BSD 3 | | wcwidth | wcwidth-0.2.5-py2.py3-none-any.whl | MIT | | @types/webpack | webpack-4.41.17.tgz | MIT | | @types/webpack-sources | webpack-sources-1.4.0.tgz | MIT | | websocket\_client | websocket\_client-0.57.0-py2.py3-none-any.whl | BSD 3 | | whatwg-fetch | whatwg-fetch-2.0.4.tgz | MIT | | worker-rpc | worker-rpc-0.1.1.tgz | MIT | | zipp | zipp-1.2.0-py2.py3-none-any.whl | MIT | | github.com/projectcalico/bird | github.com/projectcalico/bird/blob/v0.3.3 | GPL | | egress-gateway | egress-gateway | Tigera Proprietary | ### REST API Reference ### Frequently asked questions ## Why use Calico Cloud?[​](#why-use-calico-cloud) The problem Calico Cloud tries to solve is the networking of workloads (VMs, containers, etc) in a high scale environment. Existing L2-based methods for solving this problem have problems at high scale. Compared to these, we think Calico Cloud is more scalable, simpler, and more flexible. We think you should look into it if you have more than a handful of nodes on a single site. Calico Cloud also provides a rich network security model that allows operators and developers to declare intent-based network security policy that is automatically rendered into distributed firewall rules across a cluster of containers, VMs, and/or servers. For a more detailed discussion of this topic, see our blog post at [Why Calico?](https://www.projectcalico.org/why-calico/). ## Does Calico Cloud work with IPv6?[​](#does-calico-cloud-work-with-ipv6) Yes! Calico Cloud's core components support IPv6 out of the box. However, not all orchestrators that we integrate with support IPv6 yet. ## Why does my container have a route to 169.254.1.1?[​](#why-does-my-container-have-a-route-to-16925411) In a Calico Cloud network, each host acts as a gateway router for the workloads that it hosts. In container deployments, Calico Cloud uses 169.254.1.1 as the address for the Calico Cloud router. By using a link-local address, Calico Cloud saves precious IP addresses and avoids burdening the user with configuring a suitable address. While the routing table may look a little odd to someone who is used to configuring LAN networking, using explicit routes rather than subnet-local gateways is fairly common in WAN networking. ## Why isn't Calico Cloud working with a containerized Kubelet?[​](#why-isnt-calico-cloud-working-with-a-containerized-kubelet) Calico Cloud hosted install places the necessary CNI binaries and config on each Kubernetes node in a directory on the host as specified in the manifest. By default it places binaries in /opt/cni/bin and config /etc/cni/net.d. When running the kubelet as a container using hyperkube, you need to make sure that the containerized kubelet can see the CNI network plugins and config that have been installed by mounting them into the kubelet container. For example add the following arguments to the kubelet-wrapper service: ```text --volume /etc/cni/net.d:/etc/cni/net.d \ --volume /opt/cni/bin:/opt/cni/bin \ ``` Without the above volume mounts, the kubelet will not call the Calico Cloud CNI binaries, and so Calico Cloud [workload endpoints](https://docs.tigera.io/calico-cloud/reference/resources/workloadendpoint) will not be created, and Calico Cloud policy will not be enforced. ## How do I view Calico Cloud CNI logs?[​](#how-do-i-view-calico-cloud-cni-logs) The Calico Cloud CNI plugin emits logs to stderr, which are then logged out by the kubelet. Where these logs end up depend on how your kubelet is configured. For deployments using `systemd`, you can do this via `journalctl`. The log level can be configured via the CNI network configuration file, by changing the value of the key `log_level`. See [Configuring the Calico Cloud CNI plugins](https://docs.tigera.io/calico-cloud/reference/component-resources/configuration) for more information. CNI plugin logs can also be found in `/var/log/calico/cni`. ## How do I configure the pod IP range?[​](#how-do-i-configure-the-pod-ip-range) When using Calico Cloud IPAM, IP addresses are assigned from [IP Pools](https://docs.tigera.io/calico-cloud/reference/resources/ippool). By default, all enabled IP pools are used. However, you can specify which IP pools to use for IP address management in the [CNI network config](https://docs.tigera.io/calico-cloud/reference/component-resources/configuration#ipam), or on a per-pod basis using [Kubernetes annotations](https://docs.tigera.io/calico-cloud/reference/component-resources/configuration#using-kubernetes-annotations). ## How do I assign a specific IP address to a pod?[​](#how-do-i-assign-a-specific-ip-address-to-a-pod) For most use cases it's not necessary to assign specific IP addresses to a Kubernetes pod and it's recommended to use Kubernetes services instead. However, if you do need to assign a particular address to a pod, Calico Cloud provides two ways of doing this: - You can request an IP that is available in Calico Cloud IPAM using the `cni.projectcalico.org/ipAddrs` annotation. - You can request an IP using the `cni.projectcalico.org/ipAddrsNoIpam` annotation. Note that this annotation bypasses the configured IPAM plugin, and thus in most cases it is recommended to use the above annotation. See the [Requesting a specific IP address](https://docs.tigera.io/calico-cloud/reference/component-resources/configuration#requesting-a-specific-ip-address) section in the CNI plugin reference documentation for more details. ## Why can't I see the 169.254.1.1 address mentioned above on my host?[​](#why-cant-i-see-the-16925411-address-mentioned-above-on-my-host) Calico Cloud tries hard to avoid interfering with any other configuration on the host. Rather than adding the gateway address to the host side of each workload interface, Calico Cloud sets the `proxy_arp` flag on the interface. This makes the host behave like a gateway, responding to ARPs for 169.254.1.1 without having to actually allocate the IP address to the interface. ## Why do all cali\* interfaces have the MAC address ee:ee:ee:ee:ee:ee:?[​](#why-do-all-cali-interfaces-have-the-mac-address-eeeeeeeeeeee) In some setups the kernel is unable to generate a persistent MAC address and so Calico Cloud assigns a MAC address itself. Since Calico Cloud uses point-to-point routed interfaces, traffic does not reach the data link layer so the MAC Address is never used and can therefore be the same for all the cali\* interfaces. ## Can I prevent my Kubernetes pods from initiating outgoing connections?[​](#can-i-prevent-my-kubernetes-pods-from-initiating-outgoing-connections) Yes! The Kubernetes [`NetworkPolicy`](https://kubernetes.io/docs/concepts/services-networking/network-policies/) API added support for egress policies in v1.8. You can also use `calicoctl` to configure egress policy to prevent Kubernetes pods from initiating outgoing connections based on the full set of supported Calico Cloud policy primitives including labels, Kubernetes namespaces, CIDRs, and ports. ## I've heard Calico Cloud uses proxy ARP, doesn't proxy ARP cause a lot of problems?[​](#ive-heard-calico-cloud-uses-proxy-arp-doesnt-proxy-arp-cause-a-lot-of-problems) It can, but not in the way that Calico Cloud uses it. In container deployments, Calico Cloud only uses proxy ARP for resolving the 169.254.1.1 address. The routing table inside the container ensures that all traffic goes via the 169.254.1.1 gateway so that is the only IP that will be ARPed by the container. ## Is Calico Cloud compliant with PCI/DSS requirements?[​](#is-calico-cloud-compliant-with-pcidss-requirements) PCI certification applies to the whole end-to-end system, of which Calico Cloud would be a part. We understand that most current solutions use VLANs, but after studying the PCI requirements documents, we believe that Calico Cloud does meet those requirements and that nothing in the documents *mandates* the use of VLANs. ## How do I enable IP-in-IP and NAT outgoing on an IP pool?[​](#how-do-i-enable-ip-in-ip-and-nat-outgoing-on-an-ip-pool) 1. Retrieve current IP pool config. ```bash calicoctl get ipPool --export -o yaml > pool.yaml ``` 2. Modify IP pool config. Modify the pool's spec to enable IP-in-IP and NAT outgoing. (See [IP pools](https://docs.tigera.io/calico-cloud/reference/resources/ippool) for other settings that can be edited.) ```shell - apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: ippool-1 spec: cidr: 192.168.0.0/16 ipipMode: Always natOutgoing: true ``` 3. Load the modified file. ```bash kubectl replace -f pool.yaml ``` ## How does Calico Cloud maintain saved state?[​](#how-does-calico-cloud-maintain-saved-state) State is saved in a few places in a Calico Cloud deployment, depending on whether it's global or local state. Local state is state that belongs on a single compute host, associated with a single running Felix instance (things like kernel routes, tap devices etc.). Local state is entirely stored by the Linux kernel on the host, with Felix storing it only as a temporary mirror. This makes Felix effectively stateless, with the kernel acting as a backing data store on one side and Kubernetes (kdd) as a data source on the other. If Felix is restarted, it learns current local state by interrogating the kernel at start up. It then reads from the etcd datastore all the local state which it should have, and updates the kernel to match. This approach has strong resiliency benefits, in that if Felix restarts you don't suddenly lose access to your VMs or containers. As long as the Linux kernel is running, you've still got full functionality. The bulk of global state is mastered in whatever component hosts the plugin. - In certain cases, `etcd` itself contains the master copy of the data. This is because some Docker deployments have an `etcd` cluster that has the required resiliency characteristics, used to store all system configuration and so `etcd` is configured so as to be a suitable store for critical data. - In other orchestration systems, it may be stored in distributed databases, either owned directly by the plugin or by the orchestrator itself. The only other state storage in a Calico Cloud network is in the BGP sessions, which approximate a distributed database of routes. This BGP state is simply a replicated copy of the per-host routes configured by Felix based on the global state provided by the orchestrator. This makes the Calico Cloud design very simple, because we store very little state. All of our components can be shut down and restarted without risk, because they resynchronize state as necessary. This makes modeling their behavior extremely simple, reducing the complexity of bugs. ## I heard Calico Cloud is suggesting layer 2: I thought you were layer 3! What's happening?[​](#i-heard-calico-cloud-is-suggesting-layer-2-i-thought-you-were-layer-3-whats-happening) It's important to distinguish what Calico Cloud provides to the workloads hosted in a data center (a purely layer 3 network) with what the Calico Cloud project *recommends* operators use to build their underlying network fabric. Calico Cloud's core principle is that *applications* and *workloads* overwhelmingly need only IP connectivity to communicate. For this reason we build an IP-forwarded network to connect the tenant applications and workloads to each other and the broader world. However, the underlying physical fabric obviously needs to be set up too. Here, Calico Cloud has discussed how both a layer 2 (see [here](https://docs.tigera.io/calico-cloud/reference/architecture/design/l2-interconnect-fabric)) or a layer 3 (see [here](https://docs.tigera.io/calico-cloud/reference/architecture/design/l3-interconnect-fabric)) fabric could be integrated with Calico Cloud. This is one of the great strengths of the Calico Cloud model: it allows the infrastructure to be decoupled from what we show to the tenant applications and workloads. We have some thoughts on different interconnect approaches (as noted above), but just because we say that there are layer 2 and layer 3 ways of building the fabric, and that those decisions may have an impact on route scale, does not mean that Calico Cloud is "going back to Ethernet" or that we're recommending layer 2 for tenant applications. In all cases we forward on IP packets, no matter what architecture is used to build the fabric. ## How do I control policy/connectivity without virtual/physical firewalls?[​](#how-do-i-control-policyconnectivity-without-virtualphysical-firewalls) Calico Cloud provides an extremely rich security policy model, applying policy at the first and last hop of the routed traffic within the Calico Cloud network (the source and destination compute hosts). This model is substantially more robust to failure than a centralized firewall-based model. In particular, the Calico Cloud approach has no single point of failure: if the device enforcing the firewall has failed then so has one of the workloads involved in the traffic (because the firewall is enforced by the compute host). This model is also extremely amenable to scaling out. Because we have a central repository of policy configuration, but apply it at the edges of the network (the hosts) where it is needed, we automatically ensure that the rules match the topology of the data center. This allows easy scaling out, and gives us all the advantages of a single firewall (one place to manage the rules), but none of the disadvantages (single points of failure, state sharing, hairpinning of traffic, etc.). Lastly, we decouple the reachability of nodes and the policy applied to them. We use BGP to distribute the topology of the network, telling every node how to get to every endpoint in case two endpoints need to communicate. We use policy to decide *if* those two nodes should communicate, and if so, how. If policy changes and two endpoints should now communicate, where before they shouldn’t have, all we have to do is update policy: the reachability information does not change. If later they should be denied the ability to communicate, the policy is updated again, and again the reachability doesn’t have to change. ## Why isn't the `-p` flag on `docker run` working as expected?[​](#why-isnt-the--p-flag-on-docker-run-working-as-expected) The `-p` flag tells Docker to set up port mapping to connect a port on the Docker host to a port on your container via the `docker0` bridge. If a host's containers are connected to the `docker0` bridge interface, Calico Cloud would be unable to enforce security rules between workloads on the same host; all containers on the bridge would be able to communicate with one other. You can securely configure port mapping by following \[Configure outgoing NAT]\(../networking/configuring/workloads-outside-cluster.mdx. ## Can Calico Cloud containers use any IP address within a pool, even subnet network/broadcast addresses?[​](#can-calico-cloud-containers-use-any-ip-address-within-a-pool-even-subnet-networkbroadcast-addresses) Yes! Calico Cloud is fully routed, so all IP address within a Calico Cloud pool are usable as private IP addresses to assign to a workload. This means addresses commonly reserved in a L2 subnet, such as IPv4 addresses ending in .0 or .255, are perfectly okay to use. ## How do I get network traffic into and out of my Calico Cloud cluster?[​](#how-do-i-get-network-traffic-into-and-out-of-my-calico-cloud-cluster) The recommended way to get traffic to/from your Calico Cloud network is by peering to your existing data center L3 routers using BGP and by assigning globally routable IPs (public IPs) to containers that need to be accessed from the internet. This allows incoming traffic to be routed directly to your containers without the need for NAT. This flat L3 approach delivers exceptional network scalability and performance. A common scenario is for your container hosts to be on their own isolated layer 2 network, like a rack in your server room or an entire data center. Access to that network is via a router, which also is the default router for all the container hosts. If this describes your infrastructure, \[Configure outgoing NAT]\(../networking/configuring/workloads-outside-cluster.mdx explains in more detail what to do. Otherwise, if you have a layer 3 (IP) fabric, then there are detailed datacenter networking recommendations given in [Calico Cloud over IP fabrics](https://docs.tigera.io/calico-cloud/reference/architecture/design/l3-interconnect-fabric). We'd also encourage you to [get in touch](https://www.projectcalico.org/contact) to discuss your environment. ### How can I enable NAT for outgoing traffic from containers with private IP addresses?[​](#how-can-i-enable-nat-for-outgoing-traffic-from-containers-with-private-ip-addresses) If you want to allow containers with private IP addresses to be able to access the internet then you can use your data center's existing outbound NAT capabilities (typically provided by the data center's border routers). Alternatively you can use Calico Cloud's built in outbound NAT capability by enabling it on any Calico Cloud IP pool. In this case Calico Cloud will perform outbound NAT locally on the compute node on which each container is hosted. ```bash cat < natOutgoing: true EOF ``` Where `` is the CIDR of your IP pool, for example `192.168.0.0/16`. Remember: the security profile for the container will need to allow traffic to the internet as well. Refer to the appropriate guide for your orchestration system for details on how to configure policy. ### How can I enable NAT for incoming traffic to containers with private IP addresses?[​](#how-can-i-enable-nat-for-incoming-traffic-to-containers-with-private-ip-addresses) As discussed, the recommended way to get traffic to containers that need to be accessed from the internet is to give them public IP addresses and to configure Calico Cloud to peer with the data center's existing L3 routers. In cases where this is not possible then you can configure incoming NAT (also known as DNAT) on your data centers existing border routers. Alternatively you can configure incoming NAT with port mapping on the host on which the container is running on. 1. Create a new chain called `expose-ports` to hold the NAT rules. ```bash iptables -t nat -N expose-ports ``` 2. Jump to that chain from the `OUTPUT` and `PREROUTING` chains. ```bash iptables -t nat -A OUTPUT -j expose-ports iptables -t nat -A PREROUTING -j expose-ports ``` > **SUCCESS:** The `OUTPUT` chain is hit by traffic originating on the host itself; the `PREROUTING` chain is hit by traffic coming from elsewhere. 3. For each port you want to expose, add a rule to the expose-ports chain, replacing `` with the host IP that you want to use to expose the port and `` with the host port. ```bash iptables -t nat -A expose-ports -p tcp --destination \ --dport -j DNAT --to : ``` For example, you have a container to which you've assigned the `CALICO_IP` of 192.168.7.4, and you have NGINX running on port 8080 inside the container. If you want to expose this service on port 80 and your host has IP 192.0.2.1, then you could run the following commands: ```bash iptables -t nat -N expose-ports iptables -t nat -A OUTPUT -j expose-ports iptables -t nat -A PREROUTING -j expose-ports iptables -t nat -A expose-ports -p tcp --destination 192.0.2.1 --dport 80 -j DNAT --to 192.168.7.4:8080 ``` The commands will need to be run each time the host is restarted. Remember: the security profile for the container will need to allow traffic to the exposed port as well. Refer to the appropriate guide for your orchestration system for details on how to configure policy. ### Can I run Calico Cloud in a public cloud environment?[​](#can-i-run-calico-cloud-in-a-public-cloud-environment) Yes. If you are running in a public cloud that doesn't allow either L3 peering or L2 connectivity between Calico Cloud hosts then you can enable IP-in-IP in your Calico Cloud IP pool: ```bash cat < ipipMode: Always natOutgoing: true EOF ``` Calico Cloud will then route traffic between Calico Cloud hosts using IP-in-IP. For best performance in AWS, you can disable [Source/Destination Check](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig#spec) instead of using IP-in-IP or VXLAN; but only if all your instances are in the same subnet of your VPC. The setting must be `Disable` for the EC2 instance(s) to process traffic not matching the host interface IP address. This is also applicable if your cluster is spread across multiple subnets. If your cluster traffic crosses subnets, set `ipipMode` (or `vxlanMode`) to `CrossSubnet` to reduce the encapsulation overhead. Check [configuring overlay networking](https://docs.tigera.io/calico-cloud/networking/configuring/vxlan-ipip) for the details. You can disable Source/Destination Check using [Felix configuration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig), the AWS CLI, or the EC2 console. For example, using the AWS CLI: ```bash aws ec2 modify-instance-attribute --instance-id --source-dest-check "{\"Value\": false}" cat < natOutgoing: true EOF ``` ### On AWS with IP-in-IP, why do I see no connectivity between workloads or only see connectivity if I ping in both directions?[​](#on-aws-with-ip-in-ip-why-do-i-see-no-connectivity-between-workloads-or-only-see-connectivity-if-i-ping-in-both-directions) By default, AWS security groups block incoming IP-in-IP traffic. However, if an instance has recently sent some IP-in-IP traffic out when it receives some incoming IP-in-IP traffic, then AWS sees that as a response to an outgoing connection and it allows the incoming traffic. This leads to some very confusing behavior where traffic can be blocked and then suddenly start working! To resolve the issue, add a rule to your security groups that allows inbound and outbound IP-in-IP traffic (IP protocol number 4) between your hosts. ## Can Calico do IP multicast?[​](#can-calico-do-ip-multicast) Calico is a routed L3 network where each pod gets a /32. There's no broadcast domain for pods. That means that multicast doesn't just work as a side effect of broadcast. To get multicast to work, the host needs to act as a multicast gateway of some kind. Calico's architecture was designed to extend to cover that case but it's not part of the product as yet. --- ## Support and feedback ### Support and feedback ## Contact Support[​](#contact-support) You can find solutions to many common problems by following our [troubleshooting checklist](https://docs.tigera.io/calico-cloud/get-started/checklist) or by consulting the [Tigera Help Center](https://www.tigera.io/calico-support/). For everything else, you can open a support ticket. ### Paid Calico Cloud users[​](#paid-calico-cloud-users) Sign in to the [support portal](https://tigeraio.my.site.com/community/s/login/) to open a ticket. ### Free trial users[​](#free-trial-users) From the web console, click the user icon > **Contact Support**, and then complete the form. ## Provide feedback[​](#provide-feedback) We value your feedback and suggestions for improvement. Email us: [help@calicocloud.io](mailto:help@calicocloud.io). ## Support policy[​](#support-policy) For details, see our [Support policy](https://www.tigera.io/legal/calico-cloud-support-policy). ## Check the status of Calico Cloud services[​](#check-the-status-of-calico-cloud-services) Go to [Calico Cloud Status](https://status.calicocloud.io) to view the current status of our sites and services. --- ## Release notes ### Calico Cloud release notes ## February 5, 2026 (web console update) ### New features and enhancements[​](#new-features-and-enhancements) #### Projects[​](#projects) This release introduces self-service Projects, allowing you to organize managed clusters into logical groups based on team, environment, or geographic location. You can now create up to three projects, select hosting regions, and scope observability tools to specific project environments. For more information, see [Projects](https://docs.tigera.io/calico-cloud/tutorials/calico-cloud-features/projects). #### Namespaced access to dashboards[​](#namespaced-access-to-dashboards) Administrators can now provide users with namespaced access to dashboards. This ensures that users view logs and information only for the specific areas they are authorized to manage. Namespaced access to dashboards can be managed by specifying namespaces in the **View Network Sets** custom role permission. > **INFO:** Changes in this release may cause some users to lose access to dashboard data. To restore access, administrators must edit users' custom roles to add the **View Global Network Sets** permission or the **View Network Sets** permission with the appropriate namespaces. For more information, see [Create and assign custom roles](https://docs.tigera.io/calico-cloud/users/create-and-assign-custom-roles). ### Deprecated and removed features[​](#deprecated-and-removed-features) - Kibana dashboards are deprecated and will be removed in an upcoming release. During the deprecation period, you will have read-only access to Kibana dashboards. You can still [create custom dashboards](https://docs.tigera.io/calico-cloud/observability/create-custom-dashboard) using Calico Cloud's built-in dashboards. ## December 1, 2025 (version 22.1.0) ### New features and enhancements[​](#new-features-and-enhancements-1) #### Dashboards are GA[​](#dashboards-are-ga) Calico Cloud's built-in observability dashboards are now generally available. These integrated views provide cluster operators with a detailed view of network operations, including metrics on traffic flows, policy decisions, and layer 7 (HTTP) requests. You can supplement the default dashboards by creating your own custom dashboards to suit your specific needs. For more information, see [Dashboards](https://docs.tigera.io/calico-cloud/observability/dashboards). #### Enhancements[​](#enhancements) - Added various user experience improvements to dashboards in the web console. ### Bug fixes[​](#bug-fixes) - Fixed a problem with time range filters for packet captures in Service Graph. - Fixed a bug that caused incorrect operands to appear when you added ports to a network policy in the web console. #### Support for Kubernetes 1.34[​](#support-for-kubernetes-134) This release adds support for Kubernetes 1.34. ### Known issues[​](#known-issues) > **WARNING:** Pod networking fails if you install or upgrade to Calico Cloud 22.1.0 on a cluster that has the following: > > - The cluster has Windows nodes. > - The cluster's `IPPool` resource has `spec.encapsulation` set to `None`. > > Updating a cluster with Windows nodes and no encapsulation causes pod networking to fail on Windows nodes. > > This issue will be fixed in an upcoming release. ## October 21, 2025 (web console update) ### New features and enhancements[​](#new-features-and-enhancements-2) - Improved search and user experience for the **Policies** page in the Calico Cloud Free Tier web console. ### Bug fixes[​](#bug-fixes-1) - Added various user experience fixes and improvements to dashboards in the web console. ## October 6, 2025 (version 22.0.1) ### Bug fixes[​](#bug-fixes-2) - Removed an unnecessary impersonate permission for the calico-guardian ClusterRole. The guardian now uses direct permissions instead of impersonation, improving cluster security. ## September 8, 2025 (version 22.0.0) ### New features and enhancements[​](#new-features-and-enhancements-3) #### Calico Ingress Gateway is GA[​](#calico-ingress-gateway-is-ga) Calico Ingress Gateway is now supported with general availability. Calico Ingress Gateway is an enterprise-hardened, 100% upstream distribution of Envoy Gateway. Envoy Gateway is an implementation of the Kubernetes Gateway API with several extensions that provide advanced security and traffic management features. For more information, see [Configure an ingress gateway](https://docs.tigera.io/calico-cloud/networking/ingress-gateway/about-calico-ingress-gateway). #### External load balancers for Calico Ingress Gateway[​](#external-load-balancers-for-calico-ingress-gateway) This release adds customization options for specifying external load balancers for `Gateway` resources in your cluster. For more information, see [Customize gateway deployment and features](https://docs.tigera.io/calico-cloud/networking/ingress-gateway/about-calico-ingress-gateway#customize-gateway-deployment-and-features). #### Web application firewall for Calico Ingress Gateway[​](#web-application-firewall-for-calico-ingress-gateway) The release adds the ability to configure a web application firewall for Calico Ingress Gateway. For more information, see [Deploying WAF with an ingress gateway](https://docs.tigera.io/calico-cloud/threat/deploying-waf-ingress-gateway). #### Enhanced visibility for denied pre-DNAT traffic[​](#enhanced-visibility-for-denied-pre-dnat-traffic) This update delivers flow logs specifically for denied traffic by Calico pre-DNAT policies. Now, you can quickly identify and understand why certain traffic is being blocked at the earliest stage of packet processing, greatly simplifying troubleshooting and providing assurance that your pre-DNAT policies are effectively enforcing your desired access controls. For more information, see [Apply policy to forwarded traffic](https://docs.tigera.io/calico-cloud/network-policy/hosts/host-forwarded-traffic). #### HTTP header-based matching for application layer policies[​](#http-header-based-matching-for-application-layer-policies) This release includes support for HTTP header-based matching for application layer policies. You can now write deny and allow rules for L7 ingress traffic by matching values in HTTP headers. For more information, see [Global network policy](https://docs.tigera.io/calico-cloud/reference/resources/globalnetworkpolicy#httpheadermatch). #### Enhancements[​](#enhancements-1) - To support a minimal footprint and simplify resource management, the API server component and its associated resources have been moved from the `tigera-system` namespace to the `calico-system` namespace. ### Version support[​](#version-support) You can now install or upgrade to the following versions: - Calico Cloud 22 - Calico Cloud 21 - Calico Cloud 20 ## August 5, 2025 (web console update) ### New features and enhancements[​](#new-features-and-enhancements-4) #### Predefined user roles for Entra ID groups[​](#predefined-user-roles-for-entra-id-groups) We added support for granting predefined role permissions to Entra ID security groups. This allows platform administrators to bulk assign access to the Calico Cloud web console without having to create custom roles. For more information, see [Give role-based access to an Entra ID group](https://docs.tigera.io/calico-cloud/users/create-custom-role-for-entra-id-group). ## June 23, 2025 (version 21.3.0) ### New features and enhancements[​](#new-features-and-enhancements-5) #### Policy recommendations for Calico Cloud Free Tier[​](#policy-recommendations-for-calico-cloud-free-tier) We added support for policy recommendations for Calico Cloud Free Tier. The policy recommendations tool provides a set of recommended network policies based on the traffic patterns in your cluster. When you get a recommendation, you can review, stage, and test the policy before applying it to your cluster. For more information, See [Policy recommendations](https://docs.tigera.io/calico-cloud/free/overview#policy-recommendations). #### Support for WireGuard encryption between clusters[​](#support-for-wireguard-encryption-between-clusters) We added support for WireGuard encryption between federated services and endpoints in different clusters. For more information, see [Creating the cluster mesh](https://docs.tigera.io/calico-cloud/multicluster/kubeconfig). #### Improvements to flow log reporting for staged network policies[​](#improvements-to-flow-log-reporting-for-staged-network-policies) This release introduces changes to improve how staged network policies are reported in flow logs. Previously, a flow log reported the action of staged network policy rules at the time a connection was initiated. For long-lived connections, changing a staged policy did not affect the reported action. Now, flow logs report the action that represents the current policy rules. Flow logs report an action that reflects how a new connection would interact with the current staged policies. As part of this, we've also added more granular information about policies in the flow logs. For more information, see [Flow log data types](https://docs.tigera.io/calico-cloud/observability/elastic/flow/datatypes). #### Security event webhooks for Alertmanager[​](#security-event-webhooks-for-alertmanager) We added support for using webhooks to post security alerts directly to Alertmanager. For more information, see [Webhooks for security event alerts](https://docs.tigera.io/calico-cloud/threat/configuring-webhooks). #### View rule details for Web Application Firewall[​](#view-rule-details-for-web-application-firewall) You can now use the web console to view details of the default rule set used by Web Application Firewall. From the **Web Application Firewall** page, click the **Rulesets** tab to open a list of default rules. #### Enhancements[​](#enhancements-2) - Added web console support for `AdminNetworkPolicy` and `BaseAdminNetworkPolicy` tiers (view-only). - Performance and user experience improvements to custom dashboards. #### Support for Kubernetes 1.32 and 1.33[​](#support-for-kubernetes-132-and-133) This release adds support for Kubernetes 1.32 and 1.33. ### Known issues[​](#known-issues-1) - If your managed cluster includes nodes running Bottlerocket OS, you may encounter installation errors and network connectivity issues when updating to Calico Cloud 21.3.0. To prevent or resolve these issues, modify the Felix configuration resource by setting the `dnsPolicyMode` parameter to `DelayDeniedPacket`. Apply this patch either before or after updating to Calico Cloud 21.3.0: ```bash kubectl patch --type json felixconfigurations.projectcalico.org default -p '[{"op":"add","path":"/spec/dnsPolicyMode","value":"DelayDeniedPacket"}]' ``` This issue is caused by a new default value for the `dnsPolicyMode` parameter in the Felix configuration. In version 21.3.0, if not explicitly set, `dnsPolicyMode` defaults to `Inline`, which is incompatible with Bottlerocket OS and leads to the described problems. This issue will be addressed in an upcoming release. ## May 5, 2025 (web console update)[​](#may-5-2025-web-console-update) ### New features and enhancements[​](#new-features-and-enhancements-6) #### Introducing Calico Cloud Free Tier[​](#introducing-calico-cloud-free-tier) We are excited to introduce Calico Cloud Free Tier, a new offering designed to provide users of Calico Open Source with powerful observability tools at no cost. If you're running Calico Open Source, you can now connect a cluster to our cloud-based web console to visualize network traffic, understand policy interactions, and gain essential insights into your Kubernetes networking. Calico Cloud Free Tier offers a selection of core observability features from the full Calico Cloud platform, focused on helping you understand what's happening within your Calico Open Source cluster. It leverages Calico's flow logs API (introduced in Calico Open Source 3.30) to securely stream aggregated flow log data from your cluster to the Calico Cloud console. Key features include: - Dashboards. Access pre-built dashboards visualizing traffic volume, DNS activity, and network performance based on the last 24 hours of flow data. - Service Graph. Get a dynamic, topographical map of service-to-service communication and endpoint flows within your cluster, making it easy to visualize connections and troubleshoot issues. - Policy board. Inspect your network policies and staged network policies to make sure they're working as you expect. To get started, [sign up for a free Calico Cloud account](https://calicocloud.io). For more information about Calico Cloud Free Tier, see the [overview](https://docs.tigera.io/calico-cloud/free/overview) or the [quickstart guide](https://docs.tigera.io/calico-cloud/free/quickstart). ## April 29, 2025 (version 21.2.0) ### New features and enhancements[​](#new-features-and-enhancements-7) #### New pre-defined user roles[​](#new-pre-defined-user-roles) We added two new pre-defined user roles to help you manage access to Calico Cloud: - The *User Admin* role has the ability to manage team members and their assigned roles. - The *Cluster Connection Admin* role has the ability to manage cluster connections and their settings. For more information, see [Set up users](https://docs.tigera.io/calico-cloud/users/user-management). #### Improvements to Dashboards[​](#improvements-to-dashboards) This release includes various improvements to the look and performance of the dashboards. Several dashboards have been renamed to better reflect their content. - The Data Volume dashboard is now called the Traffic Volume dashboard. - The L7 dashboard is now called the HTTP Traffic dashboard. - The TCP Performance dashboard is now called the Network Performance dashboard. For more information, see [Dashboards](https://docs.tigera.io/calico-cloud/observability/dashboards). ## April 1, 2025 (version 21.1.0) ### Updated pricing structure[​](#updated-pricing-structure) We've updated our pricing structure to make it easier to understand and manage. Now we calculate usage based on your vCPU hours, rather than node hours. And we stopped charging for log data ingestion. When you upgrade, you'll notice a few changes: - The previous metrics (node hours and ingested log data) are gone. - The historical consumption data will continue to show your older metrics. For more information, see [Usage and billing](https://docs.tigera.io/calico-cloud/operations/usage-metrics). ### Changes to version availability[​](#changes-to-version-availability) Beginning with this release, users will not have access to versions of Calico Cloud that are older than the latest version at the time they joined. For example, if you start using Calico Cloud with version 21.1.0, you will not be able to install version 19.4.0 or version 20.0.0. As new versions are released, you will have access to all versions released after the version you joined with. ### Deprecated and removed features[​](#deprecated-and-removed-features-1) The Image Assurance and container threat detection features are deprecated and will be removed in a future release. Not all users will be able to use these deprecated features from this version onward: - For users who start using Calico Cloud in April 2025 or later, these feature are not available. - Legacy users, who started using Calico Cloud before April 2025, can continue to use these features until they are removed in a future release. This split availability applies also to the compliance reports feature, which was deprecated in release 20.3.0. ### Upgrade instructions for users of deprecated features[​](#upgrade-instructions-for-users-of-deprecated-features) By default, new installations and upgrades to Calico Cloud 21.1.0 will disable Image Assurance, container threat detection, and compliance reporting. Additional steps are required to enable these features for legacy users who started using Calico Cloud before April 2025. For more information, see [Install Calico Cloud](https://docs.tigera.io/calico-cloud/get-started/install-cluster) and [Upgrade Calico Cloud](https://docs.tigera.io/calico-cloud/get-started/upgrade-cluster). ## March 10, 2025 (version 21.0.0) ### New features and enhancements[​](#new-features-and-enhancements-8) #### Custom dashboards (tech preview)[​](#custom-dashboards-tech-preview) You can now build your own dashboard visualizations by creating new charts and queries in a custom dashboard. For simple changes, you can import one of the many standard charts and make edits to the query parameters as needed. Advanced users can create custom charts from scratch to display information from our flow, DNS, and L7 logs. For more information, see [Create a custom dashboard](https://docs.tigera.io/calico-cloud/observability/create-custom-dashboard). #### Introducing Calico Ingress Gateway (tech-preview)[​](#introducing-calico-ingress-gateway-tech-preview) Calico Cloud now includes the ability to deploy Calico Ingress Gateway which is an Enterprise hardened, 100% upstream distribution of Envoy Gateway. Envoy Gateway is an implementation of the Kubernetes Gateway API with several extensions that provide advanced security and traffic management features. For more information, see [Configure an ingress gateway](https://docs.tigera.io/calico-cloud/networking/ingress-gateway/about-calico-ingress-gateway). #### IPAM for load balancers[​](#ipam-for-load-balancers) Calico Cloud now extends its IPAM capabilities to support service LoadBalancer IP allocation, providing a centralized, automated approach to managing LoadBalancer IPs within Kubernetes clusters. For more information, see [LoadBalancer IP address management](https://docs.tigera.io/calico-cloud/networking/ipam/service-loadbalancer). #### Enhancements[​](#enhancements-3) - We improved how the web console performs in cases where there are a large number of policy tiers for managed clusters. To see these optimizations, make sure your managed clusters are running Calico Cloud 21.0.0 or later. Calico Cloud versions 18 and earlier are no longer supported. For information about upgrading, see [Upgrade Calico Cloud](https://docs.tigera.io/calico-cloud/get-started/upgrade-cluster). ### Deprecated and removed features[​](#deprecated-and-removed-features-2) - The admission controller feature, which had been in tech preview, was removed. ### Version support[​](#version-support-1) You can now install or upgrade to the following versions: - Calico Cloud 21 - Calico Cloud 20 - Calico Cloud 19 ## February 11, 2025 (version 20.4.0) ### New features and enhancements[​](#new-features-and-enhancements-9) #### Exporting custom dashboard cards as CSV files[​](#exporting-custom-dashboard-cards-as-csv-files) You can now export data from your custom dashboard cards as CSV files. This allows you to analyze the exported data using spreadsheet software or data analysis libraries. #### Support for Kubernetes 1.31[​](#support-for-kubernetes-131) This release adds support for Kubernetes 1.31. #### Control-plane label customization for AKS[​](#control-plane-label-customization-for-aks) We added support for customizing the namespace labels on AKS clusters. By default we apply a `control-plane` label to namespaces so that they are exempt from Azure Policy. If you wish to apply Azure Policy to our namespaces, you can now override this label. ## January 9, 2025 (web console update)[​](#january-9-2025-web-console-update) ### New features and enhancements[​](#new-features-and-enhancements-10) #### Secure web console access with IP allowlists[​](#secure-web-console-access-with-ip-allowlists) We've added support for restricting access to the web console by IP address. You can now define a list of allowed IP addresses or ranges. For example, you can allow users to access the web console only from your corporate network. To configure an IP allowlist, [open a support ticket](https://tigeraio.my.site.com/community/s/login/). ## December 19, 2024 (version 20.3.1) ### Bug fixes[​](#bug-fixes-3) - Previously, updating Calico Cloud introduced breaking changes for some users who had defined network policy tiers with orders higher than the default value. This was caused by a change to the default tier order to 1,000,000. Now, during upgrades, the installation script checks to see whether the change in default tier order will cause problems. If necessary, the script blocks the new installation and prompts the user to adjust their tier order. - Fixed an issue so that guardian picks up refreshed Service Account tokens which caused an issue on some AKS configurations due to a token expiration change. ## December 17, 2024 (web console update)[​](#december-17-2024-web-console-update) ### New features and enhancements[​](#new-features-and-enhancements-11) #### Dashboards (tech preview)[​](#dashboards-tech-preview) We added a set of dashboards to help you understand the activity in your cluster. Each dashboard is made up of graphs, charts, and diagrams that visually represent the data in your logs. You can also create your own arrangement by creating a custom dashboard. With a custom dashboard, you can combine and arrange cards from any of the other dashboards. For more information, see [Dashboards](https://docs.tigera.io/calico-cloud/observability/dashboards). ## December 3, 2024 (version 20.3.0)[​](#december-3-2024-version-2030) > **INFO:** Because of a breaking change, this version is no longer available. Upgrade instead to [Calico Cloud 20.3.1](#20.3.1) or later. ### New features and enhancements[​](#new-features-and-enhancements-12) #### Image Assurance scan result management[​](#image-assurance-scan-result-management) In this release, you can more easily manage your Image Assurance scan results by deleting results you don't need. On the **All Scan Results** page, select the checkbox next to result item, and then click **Actions > Delete**. You can also select multiple results and delete them as a bulk action. #### Support for Windows nodes on EKS[​](#support-for-windows-nodes-on-eks) We've expanded our support for Windows nodes to include hybrid clusters on Amazon Elastic Kubernetes Service. #### Search by policy name or namespace[​](#search-by-policy-name-or-namespace) Calico Cloud now includes search-to-filter capabilities on the policy board and listing pages, which helps you find a specific policy or a subset of policies more quickly. #### Envoy deployment as a sidecar[​](#envoy-deployment-as-a-sidecar) Calico Cloud now provides the ability to deploy Envoy as a sidecar so application layer policy and logging are compatible with other features such as egress gateways, Wireguard for data-in-transit encryption, and Calico’s eBPF data plane. For more information, see [Application layer policy](https://docs.tigera.io/calico-cloud/network-policy/application-layer-policies/alp) and [L7 logs](https://docs.tigera.io/calico-cloud/observability/elastic/l7/configure). #### Configurable rules for deep packet inspection[​](#configurable-rules-for-deep-packet-inspection) Calico Cloud now provides the ability for administrators to configure and customize the Snort rules that are used in deep packet inspection. This gives customers greater control over the types of rules that are evaluated. It also ensures that they can effectively tune and selectively enable rules to phase their deep packet inspection and network-based threat detection. For more information, see [Deep packet inspection](https://docs.tigera.io/calico-cloud/threat/deeppacketinspection). #### Calico early networking (for dual ToR) preserves post-boot default routes[​](#calico-early-networking-for-dual-tor-preserves-post-boot-default-routes) Calico Cloud includes improvements so that early network configuration will be superseded by any BGPPeer or BGPConfiguration resources after successful startup For more information see [Deploy a dual ToR cluster](https://docs.tigera.io/calico-cloud/networking/configuring/dual-tor). #### Enhancements[​](#enhancements-4) - Guardian will respect HTTP proxy environment variables when set on the deployment by mutating webhook configurations. - Enhanced filtering options in the endpoints page of the web console. - Performance improvements to Image Assurance processes. ### Deprecated and removed features[​](#deprecated-and-removed-features-3) - All compliance reporting features are deprecated and will be removed in a future release. We're building a new compliance reporting system that will eventually replace the current one. ### Updating[​](#updating) - ***Breaking change:*** Previously, the default tier had no order and was always evaluated last. Starting with Calico Cloud 20.3.0, the default tier now has an order of 1,000,000. When upgrading, you must ensure that existing tiers have a lower order or else policy decisions may be affected. For more information, see [Get started with policy tiers](https://docs.tigera.io/calico-cloud/network-policy/policy-tiers/tiered-policy). ## November 6, 2024 (version 20.2.0)[​](#november-6-2024-version-2020) ### New features and enhancements[​](#new-features-and-enhancements-13) #### Image Assurance scan result management[​](#image-assurance-scan-result-management-1) In this release, you can more easily manage your Image Assurance scan results by deleting results you don't need. On the **All Scan Results** page, select the checkbox next to result item, and then click **Actions > Delete**. You can also select multiple results and delete them as a bulk action. #### Enhancements[​](#enhancements-5) - Various detector improvements, including better handling of historical data and a detector export function. - Improved webhooks with ability to send global alerts. - Reduced memory usage for clusters with many `ConfigMap` resources. ### Bug fixes[​](#bug-fixes-4) - Fixed an issue with the Image Assurance CLI scanner to ensure it takes CVE exceptions into account when scanning multiple images from the command line at the same time. ## October 1, 2024 (version 20.1.0)[​](#october-1-2024-version-2010) ### New features and enhancements[​](#new-features-and-enhancements-14) #### View and manage detectors for Container Threat Detection[​](#view-and-manage-detectors-for-container-threat-detection) We've provided better access to the detectors we use as part of our Container Threat Detection system. You can now view the complete list of detectors and turn them on or off as you see fit. Detectors can also be configured as part of a new RuntimeSecurity custom resource. For more information, see [Update detector settings](https://docs.tigera.io/calico-cloud/threat/container-threat-detection#update-detectors-settings). #### Create Security Event exceptions for known processes[​](#create-security-event-exceptions-for-known-processes) We added a way to create Security Event exceptions for processes in your cluster that you know to be safe. This can be a helpful way to eliminate noise and false positives in your alerts. For more information, see [Exclude a process from Security Events alerts](https://docs.tigera.io/calico-cloud/threat/container-threat-detection#exclude-process). #### Added EPSS data to Image Assurance results[​](#added-epss-data-to-image-assurance-results) Image Assurance scans results now include information using the [Exploit Prediction Scoring System (EPSS)](https://www.first.org/epss/). EPSS scores help you determine the likelihood that a given vulnerability will be exploited in the near future. Being able to view this information and filter scan results by EPSS score can help you judge the risk of vulnerabilities and prioritize your remediation efforts. #### Enhancements[​](#enhancements-6) - Functional and performance improvements to Image Assurance scan results filtering. ## September 20, 2024 (version 20.0.1)[​](#september-20-2024-version-2001) ### Bug fixes[​](#bug-fixes-5) - Fixed an issue so that guardian picks up refreshed Service Account tokens which caused an issue on some AKS configurations due to a token expiration change. ## September 10, 2024 (version 20.0.0)[​](#september-10-2024-version-2000) ### New features and enhancements[​](#new-features-and-enhancements-15) #### Helm customizations[​](#helm-customizations) We've added new options for customizing Helm installations: - You can now enable or disable the Compliance and Packet Capture features at installation. - For many Calico Cloud components, you can specify node selectors, tolerations, and resource requests and limits. For more information, see [Connect a cluster to Calico Cloud](https://docs.tigera.io/calico-cloud/get-started/install-cluster) and [Install Calico Cloud as part of an automated workflow](https://docs.tigera.io/calico-cloud/get-started/install-automated) #### Automatic Calico Cloud access for connected IdP groups[​](#automatic-calico-cloud-access-for-connected-idp-groups) We made it easier for administrators to control access to Calico Cloud by managing users in their existing identity provider groups. When you add an identity provider, users who have memberships of one or more IdP Groups that have been enabled on Calico Cloud can log in without needing to be invited individually. To enable automatic access for your IdP group members, [open a support ticket](https://tigeraio.my.site.com/community/s/login/). #### Support for ARM64[​](#support-for-arm64) This release adds support for clusters running on ARM64 architectures. #### Support for Kubernetes 1.30[​](#support-for-kubernetes-130) This release adds support for Kubernetes 1.30. ### Deprecated and removed features[​](#deprecated-and-removed-features-4) - The honeypods feature has been removed from this release. ### Version support[​](#version-support-2) You can now install or upgrade to the following versions: - Calico Cloud 20 - Calico Cloud 19 - Calico Cloud 18 Calico Cloud versions 17 and earlier are no longer supported. For information about upgrading, see [Upgrade Calico Cloud](https://docs.tigera.io/calico-cloud/get-started/upgrade-cluster). ## July 23, 2024 (version 19.4.1)[​](#july-23-2024-version-1941) ### Bug fixes[​](#bug-fixes-6) - For AKS clusters with managed Calico, AKS changed the resources deployed which causes the installer to fail or for existing clusters the operator will stop managing resources. We made changes to the Calico Cloud installation to deploy additional resources needed to allow normal operation to continue and installs to proceed. ## July 9, 2024 (version 19.4.0)[​](#july-9-2024-version-1940) ## New features and enhancements[​](#new-features-and-enhancements-16) ### Bulk vulnerability exceptions for Image Assurance[​](#bulk-vulnerability-exceptions-for-image-assurance) We added a way to efficiently add large numbers of vulnerability exceptions to your Image Assurance scan results. Instead of creating exceptions one by one, you can add them all at once by uploading a CSV file with the vulnerability definitions. For more information, see [Exclude vulnerabilities from scan results](https://docs.tigera.io/calico-cloud/image-assurance/exclude-vulnerabilities-from-scan-results). ### Bug fixes[​](#bug-fixes-7) - Previously, defining new values for the crawdad daemon by using the ImageAssurance custom resource had no effect, and the default values remained in place. This problem is now fixed. - We fixed a bug that caused problems during Helm upgrades from Calico Cloud versions earlier than 19.1.0. ## June 11, 2024 (version 19.3.0)[​](#june-11-2024-version-1930) ### New features and enhancements[​](#new-features-and-enhancements-17) #### Jira integration for Image Assurance scan results[​](#jira-integration-for-image-assurance-scan-results) We added a way to create and assign Jira issues directly from your Image Assurance scan results page. You can filter and prioritize vulnerabilities, and then assign the remediation work to members of your team. Calico Cloud populates the information you need, including a CSV file with detailed information about the vulnerabilities in your packages. For more information, see [Creating Jira issues for scan results](https://docs.tigera.io/calico-cloud/image-assurance/creating-jira-issues-for-scan-results) #### Security events dashboard[​](#security-events-dashboard) A new dashboard summarizes security events and helps practitioners easily understand how events map across namespaces, MITRE techniques, event types, and attack phases. This allows first responders to quickly make sense of potential threats, engage the right stakeholders, and start the incident response and investigation process. For more information, see [Security event management](https://docs.tigera.io/calico-cloud/threat/security-event-management). #### Exceptions for security events[​](#exceptions-for-security-events) Calico Cloud now allows users to create exceptions for Security Events with varying levels of scope, from excluding an entire namespace to a specific deployment or workload. This gives operators a way to tune the runtime threat detection they have deployed and focus their investigations and response on critical applications and infrastructure. For more information, see [Security event management](https://docs.tigera.io/calico-cloud/threat/security-event-management). #### New flow logs panel for Endpoints and View Policy pages[​](#new-flow-logs-panel-for-endpoints-and-view-policy-pages) Calico Cloud has added new entry points to view flow logs directly from the Endpoints listing and View Policy pages in the UI. Users can easily see which endpoints are involved in denied traffic, filter on those workloads, and click a link to open a panel that shows associated flows. A similar link has been added for View Policy pages, which allows users to quickly see the flows that have been recently evaluated by that policy to make sense of denied traffic or updates to rules. #### Security Events in Service Graph[​](#security-events-in-service-graph) Calico Cloud now includes a new tab for Security Events which has taken the Alerts. Most runtime threat detection features now generate Security Events, and their inclusion Service Graph enables users to automatically filter events based on where they are occurring in a cluster. #### Security Events IP addresses enriched with ASN and geolocation[​](#security-events-ip-addresses-enriched-with-asn-and-geolocation) For security events that contain external IP addresses, Calico Cloud now automatically performs a geolocation lookup. Understanding the country of origin for an IP address can often be the quickest and easiest way to distinguish legitimate traffic from malicious traffic. #### Extend Workload-based WAF to Ingress Gateways[​](#extend-workload-based-waf-to-ingress-gateways) This latest release enables operators to plug-in a modifiedsimplified version of WAF to their own instances of Envoy. This allows users to deploy this version of WAF at the edge of their cluster integrated with an Ingress Gateway (if based on Envoy), with fully customizable rules based on OWASP CoreRuleSet 4.0 and powered by the Coraza engine. For more information, see [Deploying WAF with an ingress gateway ](https://docs.tigera.io/calico-cloud/threat/deploying-waf-ingress-gateway). #### Specifying resource requests and limits in Calico Cloud components[​](#specifying-resource-requests-and-limits-in-calico-cloud-components) Calico Cloud now provides the ability to set resource requests and limits for the components that run as part of Calico Cloud. Please see documentation for specific guidance on setting these limits. ### Known issues[​](#known-issues-2) - It is no longer supported to uninstall Calico Cloud on a cluster that before connecting to Calico Cloud it had Calico managed with AddonManager. This includes AKS clusters that had Calico installed and managed by AKS. ### Bug fixes[​](#bug-fixes-8) - Fixed an issue where occasionally the status of a cluster would not be updated when the cluster connected resulting in auth tokens and license not being updated in the cluster. ## April 30, 2024 (version 19.2.0)[​](#april-30-2024-version-1920) ### New features and enhancements[​](#new-features-and-enhancements-18) #### Automated installation with client credentials[​](#automated-installation-with-client-credentials) You can now generate and manage client credentials that you can use to automate the Calico Cloud installation process. With persistent API keys, you can build repeatable installation commands that connect your clusters as part of an automated workflow. For more information, see [Install Calico Cloud as part of an automated workflow](https://docs.tigera.io/calico-cloud/get-started/install-automated). #### Feature options for Helm installations[​](#feature-options-for-helm-installations) For Helm installations, you can now configure some feature options during installation. You can enable or disable Image Assurance, Container Threat Detection, and the Security Posture Dashboard by adding optional parameters to your Helm command. For more information, see [Connect a cluster to Calico Cloud](https://docs.tigera.io/calico-cloud/get-started/install-cluster). #### Namespace exclusions for image scanning and runtime view[​](#namespace-exclusions-for-image-scanning-and-runtime-view) We added the ability to exclude namespaces from image scanning and runtime view. By excluding certain namespaces, you can reduce noise in your scan results and focus attention on higher priority workloads. For more information, see [Configure exclusions for image scanning](https://docs.tigera.io/calico-cloud/image-assurance/scanners/cluster-scanner#configure-exclusions-for-image-scanning). ## April 2, 2024 (version 19.1.0)[​](#april-2-2024-version-1910) ### Bug fixes[​](#bug-fixes-9) - We fixed a problem that caused the Image Assurance operator to stop working when it reached its memory limit. ## February 28, 2024 (version 19.0.0)[​](#february-28-2024-version-1900) ### New features and enhancements[​](#new-features-and-enhancements-19) #### Improved flow log filtering for destination domains[​](#improved-flow-log-filtering-for-destination-domains) We’ve updated the Felix parameter (`dest_domains`) for DNS policy to make it easy to find only domain names that the deployment connected to (not all the domain names that got translated to the same IP address). For more information, see [Flow log data types](https://docs.tigera.io/calico-cloud/observability/elastic/flow/datatypes). #### New flow logs panel on Endpoints page[​](#new-flow-logs-panel-on-endpoints-page) We've updated the Endpoints page in the web console with a new flow logs panel so you can view and filter Endpoints associated with denied traffic. Flow log metadata includes the source, destination, ports, protocols, and other key forms. We've also updated the Policy Board to highlight policies with denied traffic. #### Improvements to security events dashboard[​](#improvements-to-security-events-dashboard) We've added the following improvements to the [Security events dashboard](https://docs.tigera.io/calico-cloud/threat/security-event-management): - Jira and Slack webhook integration for security event alerts By [configuring security event alerts](https://docs.tigera.io/calico-cloud/threat/configuring-webhooks), you can push security event alerts to Slack, Jira, or an external HTTP endpoint of your choice. This lets incident response and security teams to use native tools to respond to security event alerts. - Added threat feed alerts If you have implemented global threat feeds for suspicious activity (domains or suspicious IPs), alerts are now visible in the Security Overview dashboard. For more information on threat feeds, see [Trace and block suspicious IPs](https://docs.tigera.io/calico-cloud/threat/suspicious-ips). ## Deprecated and removed features[​](#deprecated-and-removed-features-5) - The AWS security groups integration is removed in this release. - The ingress log collection feature is removed in this release. ## January 31, 2024 (version 18.3.0)[​](#january-31-2024-version-1830) ### New features and enhancements[​](#new-features-and-enhancements-20) #### Assign custom roles to users automatically with Entra ID (formerly Azure AD) groups[​](#assign-custom-roles-to-users-automatically-with-entra-id-formerly-azure-ad-groups) We've added the ability to link custom roles in Calico Cloud to your organization's Entra ID groups. You can define and modify group membership in Entra ID, and Calico Cloud will automatically grant role-based access to users based on that group membership. For more information, see [Create a custom role for an Entra ID group](https://docs.tigera.io/calico-cloud/users/create-custom-role-for-entra-id-group). #### Export custom roles[​](#export-custom-roles) In this release you can export custom roles from a managed cluster and apply them to another managed cluster. Previously, if you wanted to duplicate roles in multiple clusters, you needed to create them manually for each cluster. Now there is a process to apply those roles quickly and accurately in all clusters. For more information, see [Creating and assigning custom roles](https://docs.tigera.io/calico-cloud/users/create-and-assign-custom-roles). #### Windows node support for Azure Kubernetes Service[​](#windows-node-support-for-azure-kubernetes-service) We've added support for Windows nodes in AKS clusters. #### VXLAN support for cluster mesh and federation[​](#vxlan-support-for-cluster-mesh-and-federation) We've expanded our support of cluster mesh to clusters using VXLAN for networking. Cluster mesh can be used to federate services and endpoints to authorize cross-cluster communication with Calico Cloud network policies. For more information, see [Configure federated endpoint identity and services](https://docs.tigera.io/calico-cloud/multicluster/kubeconfig). #### Support for Azure CNI with overlay networking for AKS[​](#support-for-azure-cni-with-overlay-networking-for-aks) We now officially support AKS clusters that are using overlay networking. This option is useful if you've exhausted your IP addresses. This option augments existing support for Azure CNI with no overlay (where a VNET IP address is assigned to every pod). #### Support for Kubernetes 1.28[​](#support-for-kubernetes-128) This release adds support for Kubernetes 1.28. ### Deprecated and removed features[​](#deprecated-and-removed-features-6) - The anomaly detection feature is removed in this release. If you enabled this feature, you will now stop receiving anomaly detection alerts. - The AWS security groups integration is deprecated in this release. It will be removed in a future release. - The ingress log collection feature is deprecated in this release. It will be removed in future release. ### Bug fixes[​](#bug-fixes-10) - We fixed a problem that stopped diagnostics from being collected after a failed installation. ## December 21, 2023 (version 18.2.0)[​](#december-21-2023-version-1820) ### New features and enhancements[​](#new-features-and-enhancements-21) #### Security Posture Overview dashboard[​](#security-posture-overview-dashboard) We've added a new Security Posture Overview dashboard that helps you assess the security posture of your cluster. Using a list of prioritized recommended actions, you can start to take steps to reduce your risk over time. Because the dashboard is based on existing Calico Cloud data, no configuration is required. You can start improving the security posture of your Kubernetes cluster immediately. #### Support for RKE2[​](#support-for-rke2) This release comes with support for connecting RKE2 clusters to Calico Cloud. ### Known issues[​](#known-issues-3) - You can't connect RKE2 clusters to Calico Cloud if you enabled the Image Assurance runtime view feature on any of your managed clusters. As a workaround, disable runtime view before connecting your RKE2 cluster. ### Bug fixes[​](#bug-fixes-11) - Code changes to the Kubernetes controller-runtime led to intermittent errors in how the Container Threat Detection status was displayed in the web console. We modified the Runtime Security operator to account for these changes. ## November 29, 2023 (version 18.1.0)[​](#november-29-2023-version-1810) ### New features and enhancements[​](#new-features-and-enhancements-22) - We limited the permissions that are assigned to the Calico Cloud installer. Previously, the installer had cluster administrator privileges. Now the installer gets access only to what is required to install Calico Cloud. - **Image Assurance**. We added a filter that lets you sort your list of running images by severity rating. ### Known issues[​](#known-issues-4) - If you update your cluster to a previous version of Calico Cloud (18.0.0 or earlier), you may see an erroneous message about a failed installation that took place before the successful installation. This failed installation message can be disregarded. ### Bug fixes[​](#bug-fixes-12) - Fixed an issue that caused security events generated by AKS managed clusters to be missing pod and namespace information. - Fixed an issue that caused some pods on managed clusters to crash-loop after upgrading clusters that also have Dynatrace running. ## October 23, 2023 (version 18.0.0)[​](#october-23-2023-version-1800) ### New features and enhancements[​](#new-features-and-enhancements-23) #### Image Assurance registry scanner[​](#image-assurance-registry-scanner) The Image Assurance feature adds the ability to scan images in container registries at any time, on any infrastructure, including Kubernetes. This is ideal protection for images that don’t go through a pipeline (for example, third-party images), but are published to a registry. If CVEs are missed in your build pipeline, you can catch them before they are deployed. For more information, see [Scan images in container registries](https://docs.tigera.io/calico-cloud/image-assurance/scanners/registry-scanner). #### Security event management[​](#security-event-management) We've added a new Security event management dashboard for threat detection. Security events provides context for suspicious activity detected in your cluster. Combined with the Kubernetes context, you can see what workloads are affected. For more information, see [Security event management](https://docs.tigera.io/calico-cloud/threat/security-event-management). #### New performance optimizations for egress gateways[​](#new-performance-optimizations-for-egress-gateways) Calico Cloud includes new performance options for egress gateway policies that can be used to ensure that application client and gateway pods are on the same cluster node. For more information, see [Optimize egress networking for workloads with long-lived TCP connections](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-maintenance). #### Configurable XFF headers for Envoy[​](#configurable-xff-headers-for-envoy) We've added support for XFF to propagate the original IP address when proxying application layer traffic with Envoy within a Kubernetes cluster. For more information, see [Installation reference](https://docs.tigera.io/calico-cloud/reference/installation/api#envoysettings). #### Alert-only mode for workload-based Web Application Firewall (WAF)[​](#alert-only-mode-for-workload-based-web-application-firewall-waf) We've added a new default mode for WAF that is monitor/event only. This allows operators and security teams to verify the accuracy of configured rules before actively blocking traffic. For more information, see [Web application firewall](https://docs.tigera.io/calico-cloud/threat/web-application-firewall). #### Enhancements[​](#enhancements-7) - You can now remove disconnected clusters from the list of managed clusters. See [Cluster management](https://docs.tigera.io/calico-cloud/operations/cluster-management#remove-a-cluster). ### Known issues[​](#known-issues-5) - The policy recommendation tool is not displaying policy recommendations. There is currently no workaround, but the issue will be fixed in an upcoming release. - Using a bookmarked link to log in to the Calico Cloud UI in this release causes the following problems: * Image Assurance configuration page fails to load * the web console shows the enterprise license countdown at the top of the screen even for paid customers If you are experiencing any login issues, go to [https://calicocloud.io](https://calicocloud.io) and log back in. - Calico panics if kube-proxy or other components are using native `nftables` rules instead of the `iptables-nft` compatibility shim. Until Calico supports native nftables mode, we recommend that you continue to use the iptables-nft compatibility layer for all components. (The compatibility layer was the only option before Kubernetes v1.29 added alpha-level `nftables` support.) Do not run Calico in "legacy" iptables mode on a system that is also using `nftables`. Although this combination does not panic or fail (at least on kernels that support both), the interaction between `iptables` "legacy" mode and `nftables` is confusing: both `iptables` and `nftables` rules can be executed on the same packet, leading to policy verdicts being "overturned". Note that this issue applies to all previous versions of Calico Cloud. ### Bug fixes[​](#bug-fixes-13) - We fixed an issue that caused a bad error message to appear when you changed a Calico Cloud user's role. ## September 11, 2023 (version 17.1.1)[​](#september-11-2023-version-1711) ### New features and enhancements[​](#new-features-and-enhancements-24) - We redesigned a section of the Image Assurance UI to make it easier to see how vulnerable an image is. - We made improvements to the way Container Threat Detection processes large volumes of alerts. ### Bug fixes[​](#bug-fixes-14) - We fixed a problem that caused the user interface to crash when a user attempted to edit a policy. - We fixed a problem that prevented certain images from being scanned. ## September 5, 2023 (version 17.1.0)[​](#september-5-2023-version-1710) ### New features and enhancements[​](#new-features-and-enhancements-25) #### Improvements to software versioning for Calico Cloud installations on managed clusters[​](#improvements-to-software-versioning-for-calico-cloud-installations-on-managed-clusters) We've made it easier to see what version of Calico Cloud you're running or installing on a managed cluster. Now you can: - view the Calico Cloud version number for each connected cluster from the Managed Clusters page - see when an update is available for a managed cluster - select a specific Calico Cloud version to install when you connect a cluster #### Security Events UI page[​](#security-events-ui-page) Alerts corresponding to detections generated by container threat detection will now be published to the Security Events UI page, found within the Threat Defense left navigation menu item. On this page, for every security event detected, users can view: - security event name - security event type - severity level - a description of what suspicious activity has been detected - impacted assets (pod and namespace) - attack vector type - Mitre tactic and techniques associated with the detection - mitigation recommendation - additional metadata and context associated with the detection. Alerts will continue to also appear on the Alerts UI page. For more information, see [Security event management](https://docs.tigera.io/calico-cloud/threat/security-event-management). ### Bug fixes[​](#bug-fixes-15) - Runtime Security alerts now correctly show the generated\_time as the time the alert was generated. Previously they incorrectly showed the time when the underlying event which caused the alert was generated. - Runtime Security alerts now correctly show the associated Mitre information. ### Known issues[​](#known-issues-6) - Enabling WAF and Container Threat Detection through the UI is not possible for clusters running Kubernetes v1.27+. Both features can be enabled using kubectl. - If you connected your cluster to Calico Cloud using Helm before the release of version 17.1.0, reinstalling or upgrading to any version of Calico Cloud may result in an error: "Error: rendered manifests contain a resource that already exists." Previously, the `installers.operator.calicocloud.io` custom resource definition (CRD) installed by Helm required manual upgrades. After the release of Calico Cloud 17.1.0, this CRD is updated automatically, but this change causes errors the first time you attempt to reinstall or upgrade Calico Cloud on a cluster that was connected using Helm before the release of Calico Cloud 17.1.0. As a workaround, label the CRD so that it is managed by Helm by running the following command: ```bash kubectl patch crd installers.operator.calicocloud.io -p '{"metadata": {"annotations": {"meta.helm.sh/release-name":"calico-cloud-crds","meta.helm.sh/release-namespace":"calico-cloud"},"labels":{"app.kubernetes.io/managed-by":"Helm"}}}' ``` This allows you to successfully reinstall or upgrade to Calico Cloud by following the procedure in [Upgrade Calico Cloud](https://docs.tigera.io/calico-cloud/get-started/upgrade-cluster). ### Security updates[​](#security-updates) - Runtime security upgraded to [golang 1.20.7](https://go.dev/doc/devel/release#go1.20.7), which includes security updates. - We rebuilt `cc-operator` and `cc-cni-config-scanner`, which has reduced the number of CVEs. ## August 21, 2023 (version 17.0.0)[​](#august-21-2023-version-1700) ### New features and enhancements[​](#new-features-and-enhancements-26) #### New policy recommendations engine for namespace isolation[​](#new-policy-recommendations-engine-for-namespace-isolation) Calico Cloud has added a new policy recommendations engine that automatically generates staged policies for namespace isolation within your cluster. [Policy recommendations](https://docs.tigera.io/calico-cloud/network-policy/recommendations/policy-recommendations). #### Destination-based routing for egress gateways[​](#destination-based-routing-for-egress-gateways) Calico Cloud introduces a new mode for egress gateways that can leverage destination-based routing. Destination-based routing for egress gateways allows operators associated with a destination that is external to a Kubernetes cluster (for example, IP address or CIDR), to a specific egress gateway deployment. [Egress gateways](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-on-prem). #### Support for DNS rules in clusters using NodeLocal DNSCache[​](#support-for-dns-rules-in-clusters-using-nodelocal-dnscache) Calico Cloud has added support for DNS rules in clusters using NodeLocal DNSCache. Also related, there is new documentation on using Calico policy to secure DNS traffic within the cluster with NodeLocal DNSCache enabled. [Use NodeLocal DNSCache in your cluster](https://docs.tigera.io/calico-cloud/networking/configuring/node-local-dns-cache). #### Improved UI for configuring Workload-based Web Application Firewall (WAF)[​](#improved-ui-for-configuring-workload-based-web-application-firewall-waf) Calico Cloud includes updates to the UI that allows you to select which services are enabled for the Workload-based Web Application Firewall. [Web application firewall](https://docs.tigera.io/calico-cloud/threat/web-application-firewall). #### Wireguard support for AKS and EKS with Calico CNI[​](#wireguard-support-for-aks-and-eks-with-calico-cni) Calico Cloud now offers official support for Wireguard when using Microsoft AKS or Amazon EKS with Calico CNI. This mode of deployment offers performance benefits and a more efficient routing table compared to using cloud provider CNIs. [Encrypt data in transit](https://docs.tigera.io/calico-cloud/compliance/encrypt-cluster-pod-traffic). #### Additional custom roles for Calico Cloud[​](#additional-custom-roles-for-calico-cloud) You can now create custom role-based access controls for two new roles: "Usage Metrics" and "Image Assurance Admin". #### Image Assurance improvements[​](#image-assurance-improvements) - A containerized version of Image Assurance scanner is now available to integrate into your CI/CD platform. See [Image Assurance containerized scanner](https://quay.io/repository/tigera/image-assurance-scanner-cli) to pull the latest image. - Substantial UI improvements including a new package-centric view of images ### Known issues[​](#known-issues-7) - The canvas on Service Graph may zoom and pan unexpectedly when modifying Views or Layers - Dragging tiers to modify their order is currently not working in the UI, though you can still change its order when editing a tier - Policy recommendations may generate rules with ports and protocols for intra-namespace traffic. This will be modified in the next patch release to exclude ports and protocols and provide an option to Allow or Pass this traffic. ## June 6, 2023[​](#june-6-2023) ### New features and enhancements[​](#new-features-and-enhancements-27) #### In-cluster scanning with Image Assurance[​](#in-cluster-scanning-with-image-assurance) Calico Cloud now includes the ability to scan and monitor the images running in your Kubernetes clusters for new vulnerabilities. In-cluster scanning will scan any new images not previously scanned, and continuously monitor the BOM (Bill of Materials) for running images that have prior scan results. #### New detectors for container threat detection[​](#new-detectors-for-container-threat-detection) We've added several new detectors for container threat detection. These detectors help identify unsanctioned use of network tools, task scheduling, container admin and Docker commands, and much more. Calico Cloud now includes over 40 different detectors across each category of the [MITRE ATT\&CK Matrix](https://attack.mitre.org/). ## May 2, 2023[​](#may-2-2023) This release includes a number of performance improvements and bug fixes. ## April 24, 2023[​](#april-24-2023) ### Depreciated support for RKE, RKE2[​](#depreciated-support-for-rke-rke2) Calico Cloud no longer supports installation on RKE or RKE2. ## April 11, 2023[​](#april-11-2023) ## New features and enhancements[​](#new-features-and-enhancements-28) ### Updates to Managed Clusters[​](#updates-to-managed-clusters) The **Managed Clusters** page has been redesigned to make it easier and more intuitive to search and filter your clusters. #### Egress gateways for AKS and Azure[​](#egress-gateways-for-aks-and-azure) Calico Cloud adds egress gateway support for Microsoft Azure and AKS. Egress gateways allow you to identify the namespaces associated with egress traffic outside of your cluster. [Egress gateways for AKS and Azure](https://docs.tigera.io/calico-cloud/networking/egress/egress-gateway-azure). #### UI for workload-based Web Application Firewall (WAF)[​](#ui-for-workload-based-web-application-firewall-waf) Calico Cloud includes a new UI to enable and configure a workload-based Web Application Firewall. For more information, see [Workload-based web application firewall](https://docs.tigera.io/calico-cloud/threat/web-application-firewall#enable-waf). #### Application layer policy with Envoy[​](#application-layer-policy-with-envoy) Calico Cloud now includes support for application layer policy with Envoy, enabling platform operators to define authorization rules in Calico Cloud policies for protocols such as HTTP and gRPC. For more information, see [Application layer policies](https://docs.tigera.io/calico-cloud/network-policy/application-layer-policies/). #### Service Graph performance optimizations[​](#service-graph-performance-optimizations) Calico Cloud added several optimizations to improve the performance of Service Graph for clusters with larger numbers of namespaces. #### Improvements to Envoy to accommodate advanced ingress controllers[​](#improvements-to-envoy-to-accommodate-advanced-ingress-controllers) Calico Cloud improves its Envoy deployment so you can use this feature in clusters with ingress controllers that perform advanced load balancing. For more information, see [Workload-based web application firewall](https://docs.tigera.io/calico-cloud/threat/web-application-firewall). #### Improved Calico Cloud component security[​](#improved-calico-cloud-component-security) Calico Cloud components were updated with more restrictive access for pods and containers using the Kubernetes security context: - Non-root context whenever possible - Root context and privilege escalation are used only when necessary - Added `drop ALL capabilities` for pod security - Enabled `RuntimeDefault` as the default seccomp profile for all workloads ## February 28, 2023[​](#february-28-2023) ### New features and enhancements[​](#new-features-and-enhancements-29) - Adds Bottlerocket support for Container Threat Detection. - Adds support for scanning multiple images with Image Assurance ### Bug fixes[​](#bug-fixes-16) - Fixes "Kibana" menu item rename to "Logs". - Bug fixes for Container Threat Detection alerts. ## February 7, 2023[​](#february-7-2023) ### New features and enhancements[​](#new-features-and-enhancements-30) #### New and improved Dashboards[​](#new-and-improved-dashboards) Calico Cloud includes new and improved Dashboards that enable operators to define cluster\* and namespace-scoped dashboards with new modules for policy usage, application layer and DNS metrics, and much more. #### Configure Threat Feeds in the Calico Cloud UI[​](#configure-threat-feeds-in-the-calico-cloud-ui) Calico Cloud includes a new UI that can be used to manage and configure global threat feeds For more information, see [Trace and block suspicious IPs](https://docs.tigera.io/calico-cloud/threat/suspicious-ips). #### Namespace-based policy recommendations[​](#namespace-based-policy-recommendations) Calico Cloud has improved its policy recommendation engine to add namespace-based recommendations. This enables operators to easily implement microsegmentation for namespaces. For more information, see [Create policy recommendation](https://docs.tigera.io/calico-cloud/network-policy/recommendations/policy-recommendations). #### Create custom roles for Calico Cloud users[​](#create-custom-roles-for-calico-cloud-users) Calico Cloud administrators can now define granular roles and permissions for users using custom role-based access controls. For more information, see [Create and assign custom roles](https://docs.tigera.io/calico-cloud/users/create-and-assign-custom-roles). #### Egress gateway improvements[​](#egress-gateway-improvements) Calico Cloud has improved the probes to check readiness and outbound connectivity of egress gateways. Calico Cloud has also rearchitected egress gateway pods to improve security and make use of a temporary init container to set up packet forwarding. #### Image Assurance updates[​](#image-assurance-updates) - CLI Version v1.3.4 - Calico Cloud supports the Image Assurance CLI scanner versions 1.3.0 and later. - **Bug fix:** Previously, the scanner returned an error if it reached a size limit while uploading vulnerabilities. This size limit has been removed. ## December 13, 2022[​](#december-13-2022) ### New features and enhancements[​](#new-features-and-enhancements-31) #### Search by CVE in Image Assurance[​](#search-by-cve-in-image-assurance) Image Assurance reporting features now includes a search and filtering capability that allows you to find list items based on a single CVE ID within any Image Assurance reports. #### Enable and disable Container Threat Detection in the Calico Cloud UI[​](#enable-and-disable-container-threat-detection-in-the-calico-cloud-ui) You can now enable or disable Container Threat Detection within the UI. After enabling the feature, you can review the status of which nodes are being monitored by the feature and which nodes of your cluster are unsupported. #### New Feature: Calico Cloud Service Status Page[​](#new-feature-calico-cloud-service-status-page) All users can view the status and health of the Calico Cloud service on our new status page: [https://status.calicocloud.io](https://status.calicocloud.io/). ## November 1, 2022[​](#november-1-2022) ### Image Assurance[​](#image-assurance) CLI Version v1.1.2. New CLI will now check that it is compatible with the latest Image Assurance API. ### Container Threat Detection[​](#container-threat-detection) ![tech-preview](data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnhsaW5rPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rIiB3aWR0aD0iMTAzIiBoZWlnaHQ9IjMzIiBzdHlsZT0iYmFja2dyb3VuZC1jb2xvcjojZmZmIiB2aWV3Qm94PSItMC41IC0wLjUgMTAzIDMzIj48ZGVmcz48c3R5bGU+QGltcG9ydCB1cmwoaHR0cHM6Ly9mb250cy5nb29nbGVhcGlzLmNvbS9jc3M/ZmFtaWx5PUJhcmxvdyk7PC9zdHlsZT48L2RlZnM+PHBhdGggZD0iTS0uMzQtLjMzSDEwMVYzMUgtLjM0eiIgcG9pbnRlci1ldmVudHM9ImFsbCIvPjxzd2l0Y2ggdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLS41IC0uNSkiPjxmb3JlaWduT2JqZWN0IHdpZHRoPSIxMDAlIiBoZWlnaHQ9IjEwMCUiIHBvaW50ZXItZXZlbnRzPSJub25lIiByZXF1aXJlZEZlYXR1cmVzPSJodHRwOi8vd3d3LnczLm9yZy9UUi9TVkcxMS9mZWF0dXJlI0V4dGVuc2liaWxpdHkiIHN0eWxlPSJvdmVyZmxvdzp2aXNpYmxlO3RleHQtYWxpZ246bGVmdCI+PGRpdiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94aHRtbCIgc3R5bGU9ImRpc3BsYXk6ZmxleDthbGlnbi1pdGVtczp1bnNhZmUgY2VudGVyO2p1c3RpZnktY29udGVudDp1bnNhZmUgY2VudGVyO3dpZHRoOjk5cHg7aGVpZ2h0OjFweDtwYWRkaW5nLXRvcDoxNXB4O21hcmdpbi1sZWZ0OjFweCI+PGRpdiBkYXRhLWRyYXdpby1jb2xvcnM9ImNvbG9yOiAjRkZEOTY2OyIgc3R5bGU9ImJveC1zaXppbmc6Ym9yZGVyLWJveDtmb250LXNpemU6MDt0ZXh0LWFsaWduOmNlbnRlciI+PGRpdiBzdHlsZT0iZGlzcGxheTppbmxpbmUtYmxvY2s7Zm9udC1zaXplOjEzcHg7Zm9udC1mYW1pbHk6SGVsdmV0aWNhO2NvbG9yOiNmZmQ5NjY7bGluZS1oZWlnaHQ6MS4yO3BvaW50ZXItZXZlbnRzOmFsbDtmb250LXdlaWdodDo3MDA7d2hpdGUtc3BhY2U6bm9ybWFsO292ZXJmbG93LXdyYXA6bm9ybWFsIj5UZWNoIFByZXZpZXc8L2Rpdj48L2Rpdj48L2Rpdj48L2ZvcmVpZ25PYmplY3Q+PHRleHQgeD0iNTAiIHk9IjE5IiBmaWxsPSIjRkZEOTY2IiBmb250LWZhbWlseT0iSGVsdmV0aWNhIiBmb250LXNpemU9IjEzIiBmb250LXdlaWdodD0iYm9sZCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+VGVjaCBQcmV2aWV3PC90ZXh0Pjwvc3dpdGNoPjxzd2l0Y2g+PGcgcmVxdWlyZWRGZWF0dXJlcz0iaHR0cDovL3d3dy53My5vcmcvVFIvU1ZHMTEvZmVhdHVyZSNFeHRlbnNpYmlsaXR5Ii8+PGEgeGxpbms6aHJlZj0iaHR0cHM6Ly93d3cuZGlhZ3JhbXMubmV0L2RvYy9mYXEvc3ZnLWV4cG9ydC10ZXh0LXByb2JsZW1zIiB0YXJnZXQ9Il9ibGFuayIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMCAtNSkiPjx0ZXh0IHg9IjUwJSIgeT0iMTAwJSIgZm9udC1zaXplPSIxMCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+VGV4dCBpcyBub3QgU1ZHIC0gY2Fubm90IGRpc3BsYXk8L3RleHQ+PC9hPjwvc3dpdGNoPjwvc3ZnPg==) Release of Container Threat Detection With Container Threat Detection, you can monitor container activity using eBPF. Enable this feature to receive alerts based on file and process activity for known malicious and suspicious behavior. Alert events can be viewed on the Alerts page in the web console. To get started, see [Container Threat Detection](https://docs.tigera.io/calico-cloud/threat/container-threat-detection) ## September 26, 2022[​](#september-26-2022) ### New feature: Helm[​](#new-feature-helm) Calico Cloud now supports [installation using Helm](https://docs.tigera.io/calico-cloud/get-started/install-cluster). ### New feature: Private Registry[​](#new-feature-private-registry) Calico Cloud now supports [installation from private registries](https://docs.calicocloud.io/get-started/connect/install-cluster). Note that this is only supported when installing with Helm. ### Expanded platform support: RKEv2[​](#expanded-platform-support-rkev2) Installation works on clusters with Calico deployed by RKEv2. ## September 12, 2022[​](#september-12-2022) ### Image Assurance is GA[​](#image-assurance-is-ga) Image Assurance is now released for general availability. With Image Assurance, DevOps and platform teams can scan images in public and private registries, including images that are automatically discovered in connected clusters. Image Assurance provides a runtime view into risk, based on known vulnerabilities. It also offers admission controller policies to block resources in Kubernetes from creating containers with vulnerable images from entering your cluster. #### Changes from the tech preview version[​](#changes-from-the-tech-preview-version) **New Image Assurance CLI scanner** Image scanning is now configured and performed by the `tigera-scanner` CLI. You can integrate `tigera-scanner` into your CI/CD pipelines to ensure builds are checked by Image Assurance before deployment. You can also use the CLI scanner offline and on-demand for ad hoc scanning and emergency patching. **Export options for vulnerability scan results and runtime views** We've made it easier for platform operators to share Image Assurance scan results and runtime views with these export options: - Export one row per image or one row per image and CVE. - Export CSV or JSON files. To get started, see [Image Assurance](https://docs.tigera.io/calico-cloud/image-assurance). ### Malware detection is GA[​](#malware-detection-is-ga) Malware detection is now released for general availability. Calico Cloud's malware detection identifies malicious files in your cluster and generates alerts. Calico Cloud uses eBPF-based monitoring to log file hashes of programs running in your cluster. If there's a match to known malware from our threat intelligence library, you receive an alert. You can view your alerts on the *Alerts* page on the web console. To get started, see [Malware Detection](https://docs.tigera.io/calico-cloud/threat/container-threat-detection)) ## July 27, 2022[​](#july-27-2022) Improvement: Export logs to a SIEM To help meet your compliance requirements, we've added documentation to export logs to a SIEM (syslog, Splunk, or Amazon S3). See [Export logs to a SIEM](https://docs.tigera.io/calico-cloud/observability/elastic/archive-storage). ## July 7, 2022[​](#july-7-2022) ### New feature: Distributed Web Application Firewall (WAF) with Envoy[​](#new-feature-distributed-web-application-firewall-waf-with-envoy) ![tech-preview](data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnhsaW5rPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rIiB3aWR0aD0iMTAzIiBoZWlnaHQ9IjMzIiBzdHlsZT0iYmFja2dyb3VuZC1jb2xvcjojZmZmIiB2aWV3Qm94PSItMC41IC0wLjUgMTAzIDMzIj48ZGVmcz48c3R5bGU+QGltcG9ydCB1cmwoaHR0cHM6Ly9mb250cy5nb29nbGVhcGlzLmNvbS9jc3M/ZmFtaWx5PUJhcmxvdyk7PC9zdHlsZT48L2RlZnM+PHBhdGggZD0iTS0uMzQtLjMzSDEwMVYzMUgtLjM0eiIgcG9pbnRlci1ldmVudHM9ImFsbCIvPjxzd2l0Y2ggdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLS41IC0uNSkiPjxmb3JlaWduT2JqZWN0IHdpZHRoPSIxMDAlIiBoZWlnaHQ9IjEwMCUiIHBvaW50ZXItZXZlbnRzPSJub25lIiByZXF1aXJlZEZlYXR1cmVzPSJodHRwOi8vd3d3LnczLm9yZy9UUi9TVkcxMS9mZWF0dXJlI0V4dGVuc2liaWxpdHkiIHN0eWxlPSJvdmVyZmxvdzp2aXNpYmxlO3RleHQtYWxpZ246bGVmdCI+PGRpdiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94aHRtbCIgc3R5bGU9ImRpc3BsYXk6ZmxleDthbGlnbi1pdGVtczp1bnNhZmUgY2VudGVyO2p1c3RpZnktY29udGVudDp1bnNhZmUgY2VudGVyO3dpZHRoOjk5cHg7aGVpZ2h0OjFweDtwYWRkaW5nLXRvcDoxNXB4O21hcmdpbi1sZWZ0OjFweCI+PGRpdiBkYXRhLWRyYXdpby1jb2xvcnM9ImNvbG9yOiAjRkZEOTY2OyIgc3R5bGU9ImJveC1zaXppbmc6Ym9yZGVyLWJveDtmb250LXNpemU6MDt0ZXh0LWFsaWduOmNlbnRlciI+PGRpdiBzdHlsZT0iZGlzcGxheTppbmxpbmUtYmxvY2s7Zm9udC1zaXplOjEzcHg7Zm9udC1mYW1pbHk6SGVsdmV0aWNhO2NvbG9yOiNmZmQ5NjY7bGluZS1oZWlnaHQ6MS4yO3BvaW50ZXItZXZlbnRzOmFsbDtmb250LXdlaWdodDo3MDA7d2hpdGUtc3BhY2U6bm9ybWFsO292ZXJmbG93LXdyYXA6bm9ybWFsIj5UZWNoIFByZXZpZXc8L2Rpdj48L2Rpdj48L2Rpdj48L2ZvcmVpZ25PYmplY3Q+PHRleHQgeD0iNTAiIHk9IjE5IiBmaWxsPSIjRkZEOTY2IiBmb250LWZhbWlseT0iSGVsdmV0aWNhIiBmb250LXNpemU9IjEzIiBmb250LXdlaWdodD0iYm9sZCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+VGVjaCBQcmV2aWV3PC90ZXh0Pjwvc3dpdGNoPjxzd2l0Y2g+PGcgcmVxdWlyZWRGZWF0dXJlcz0iaHR0cDovL3d3dy53My5vcmcvVFIvU1ZHMTEvZmVhdHVyZSNFeHRlbnNpYmlsaXR5Ii8+PGEgeGxpbms6aHJlZj0iaHR0cHM6Ly93d3cuZGlhZ3JhbXMubmV0L2RvYy9mYXEvc3ZnLWV4cG9ydC10ZXh0LXByb2JsZW1zIiB0YXJnZXQ9Il9ibGFuayIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMCAtNSkiPjx0ZXh0IHg9IjUwJSIgeT0iMTAwJSIgZm9udC1zaXplPSIxMCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+VGV4dCBpcyBub3QgU1ZHIC0gY2Fubm90IGRpc3BsYXk8L3RleHQ+PC9hPjwvc3dpdGNoPjwvc3ZnPg==) Calico Cloud now includes the option to enable Web Application Firewall (WAF) rule sets when using Envoy as a daemonset. This enables operators to implement an additional layer of security and threat detection for application layer traffic. See [Workload-based Web Application Firewall (WAF)](https://docs.tigera.io/calico-cloud/threat/web-application-firewall). ### New Feature: Configuration option to use DNS rules with StagedNetworkPolicies[​](#new-feature-configuration-option-to-use-dns-rules-with-stagednetworkpolicies) Calico Cloud has added a new configuration option in Felix (`DNSPolicyMode`) that lets you audit DNS rules with StagedNetworkPolicies. There is a small performance trade off if you enable this option, so we recommended to disabling it when it’s not required. See [Felix configuration](https://docs.tigera.io/calico-cloud/reference/resources/felixconfig#dnspolicymode). ### Improvement: Additional predefined RBAC options[​](#improvement-additional-predefined-rbac-options) Calico Cloud now supports 3 more pre-defined RBAC controls (devops, security and compliance persona) for role assignment. ### Improvement: Anomaly detection deployment[​](#improvement-anomaly-detection-deployment) ![tech-preview](data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnhsaW5rPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rIiB3aWR0aD0iMTAzIiBoZWlnaHQ9IjMzIiBzdHlsZT0iYmFja2dyb3VuZC1jb2xvcjojZmZmIiB2aWV3Qm94PSItMC41IC0wLjUgMTAzIDMzIj48ZGVmcz48c3R5bGU+QGltcG9ydCB1cmwoaHR0cHM6Ly9mb250cy5nb29nbGVhcGlzLmNvbS9jc3M/ZmFtaWx5PUJhcmxvdyk7PC9zdHlsZT48L2RlZnM+PHBhdGggZD0iTS0uMzQtLjMzSDEwMVYzMUgtLjM0eiIgcG9pbnRlci1ldmVudHM9ImFsbCIvPjxzd2l0Y2ggdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLS41IC0uNSkiPjxmb3JlaWduT2JqZWN0IHdpZHRoPSIxMDAlIiBoZWlnaHQ9IjEwMCUiIHBvaW50ZXItZXZlbnRzPSJub25lIiByZXF1aXJlZEZlYXR1cmVzPSJodHRwOi8vd3d3LnczLm9yZy9UUi9TVkcxMS9mZWF0dXJlI0V4dGVuc2liaWxpdHkiIHN0eWxlPSJvdmVyZmxvdzp2aXNpYmxlO3RleHQtYWxpZ246bGVmdCI+PGRpdiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94aHRtbCIgc3R5bGU9ImRpc3BsYXk6ZmxleDthbGlnbi1pdGVtczp1bnNhZmUgY2VudGVyO2p1c3RpZnktY29udGVudDp1bnNhZmUgY2VudGVyO3dpZHRoOjk5cHg7aGVpZ2h0OjFweDtwYWRkaW5nLXRvcDoxNXB4O21hcmdpbi1sZWZ0OjFweCI+PGRpdiBkYXRhLWRyYXdpby1jb2xvcnM9ImNvbG9yOiAjRkZEOTY2OyIgc3R5bGU9ImJveC1zaXppbmc6Ym9yZGVyLWJveDtmb250LXNpemU6MDt0ZXh0LWFsaWduOmNlbnRlciI+PGRpdiBzdHlsZT0iZGlzcGxheTppbmxpbmUtYmxvY2s7Zm9udC1zaXplOjEzcHg7Zm9udC1mYW1pbHk6SGVsdmV0aWNhO2NvbG9yOiNmZmQ5NjY7bGluZS1oZWlnaHQ6MS4yO3BvaW50ZXItZXZlbnRzOmFsbDtmb250LXdlaWdodDo3MDA7d2hpdGUtc3BhY2U6bm9ybWFsO292ZXJmbG93LXdyYXA6bm9ybWFsIj5UZWNoIFByZXZpZXc8L2Rpdj48L2Rpdj48L2Rpdj48L2ZvcmVpZ25PYmplY3Q+PHRleHQgeD0iNTAiIHk9IjE5IiBmaWxsPSIjRkZEOTY2IiBmb250LWZhbWlseT0iSGVsdmV0aWNhIiBmb250LXNpemU9IjEzIiBmb250LXdlaWdodD0iYm9sZCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+VGVjaCBQcmV2aWV3PC90ZXh0Pjwvc3dpdGNoPjxzd2l0Y2g+PGcgcmVxdWlyZWRGZWF0dXJlcz0iaHR0cDovL3d3dy53My5vcmcvVFIvU1ZHMTEvZmVhdHVyZSNFeHRlbnNpYmlsaXR5Ii8+PGEgeGxpbms6aHJlZj0iaHR0cHM6Ly93d3cuZGlhZ3JhbXMubmV0L2RvYy9mYXEvc3ZnLWV4cG9ydC10ZXh0LXByb2JsZW1zIiB0YXJnZXQ9Il9ibGFuayIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMCAtNSkiPjx0ZXh0IHg9IjUwJSIgeT0iMTAwJSIgZm9udC1zaXplPSIxMCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+VGV4dCBpcyBub3QgU1ZHIC0gY2Fubm90IGRpc3BsYXk8L3RleHQ+PC9hPjwvc3dpdGNoPjwvc3ZnPg==) Calico Cloud has made the configuration and deployment of anomaly detection jobs for threat detection and performance hotspots more granular, allowing you to selectively enable jobs depending on your use case. ### Improvement: the web console now displays cluster installation progress and streaming logs[​](#improvement-the-web-console-now-displays-cluster-installation-progress-and-streaming-logs) Calico Cloud now displays information about managed cluster install progress right in the UI. After you run the install command (**Connect Cluster** wizard in Managed Clusters), installation progress is automatically displayed along with logs for the managed cluster. ## May 10, 2022[​](#may-10-2022) ### New feature: Visibility into usage metrics[​](#new-feature-visibility-into-usage-metrics) Calico Cloud now displays information about cloud usage metrics. This will provide visibility into the node hours and data ingested for consumption-based invoices. Account owners can click the new "Usage Metrics" button at the bottom of the left navbar to navigate to the new page. ### Expanded platform support: AKS with managed Calico[​](#expanded-platform-support-aks-with-managed-calico) Installation works on clusters with Calico deployed by AKS. ## April 26, 2022[​](#april-26-2022) ### New feature: Malware detection[​](#new-feature-malware-detection) ![tech-preview](data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnhsaW5rPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rIiB3aWR0aD0iMTAzIiBoZWlnaHQ9IjMzIiBzdHlsZT0iYmFja2dyb3VuZC1jb2xvcjojZmZmIiB2aWV3Qm94PSItMC41IC0wLjUgMTAzIDMzIj48ZGVmcz48c3R5bGU+QGltcG9ydCB1cmwoaHR0cHM6Ly9mb250cy5nb29nbGVhcGlzLmNvbS9jc3M/ZmFtaWx5PUJhcmxvdyk7PC9zdHlsZT48L2RlZnM+PHBhdGggZD0iTS0uMzQtLjMzSDEwMVYzMUgtLjM0eiIgcG9pbnRlci1ldmVudHM9ImFsbCIvPjxzd2l0Y2ggdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLS41IC0uNSkiPjxmb3JlaWduT2JqZWN0IHdpZHRoPSIxMDAlIiBoZWlnaHQ9IjEwMCUiIHBvaW50ZXItZXZlbnRzPSJub25lIiByZXF1aXJlZEZlYXR1cmVzPSJodHRwOi8vd3d3LnczLm9yZy9UUi9TVkcxMS9mZWF0dXJlI0V4dGVuc2liaWxpdHkiIHN0eWxlPSJvdmVyZmxvdzp2aXNpYmxlO3RleHQtYWxpZ246bGVmdCI+PGRpdiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94aHRtbCIgc3R5bGU9ImRpc3BsYXk6ZmxleDthbGlnbi1pdGVtczp1bnNhZmUgY2VudGVyO2p1c3RpZnktY29udGVudDp1bnNhZmUgY2VudGVyO3dpZHRoOjk5cHg7aGVpZ2h0OjFweDtwYWRkaW5nLXRvcDoxNXB4O21hcmdpbi1sZWZ0OjFweCI+PGRpdiBkYXRhLWRyYXdpby1jb2xvcnM9ImNvbG9yOiAjRkZEOTY2OyIgc3R5bGU9ImJveC1zaXppbmc6Ym9yZGVyLWJveDtmb250LXNpemU6MDt0ZXh0LWFsaWduOmNlbnRlciI+PGRpdiBzdHlsZT0iZGlzcGxheTppbmxpbmUtYmxvY2s7Zm9udC1zaXplOjEzcHg7Zm9udC1mYW1pbHk6SGVsdmV0aWNhO2NvbG9yOiNmZmQ5NjY7bGluZS1oZWlnaHQ6MS4yO3BvaW50ZXItZXZlbnRzOmFsbDtmb250LXdlaWdodDo3MDA7d2hpdGUtc3BhY2U6bm9ybWFsO292ZXJmbG93LXdyYXA6bm9ybWFsIj5UZWNoIFByZXZpZXc8L2Rpdj48L2Rpdj48L2Rpdj48L2ZvcmVpZ25PYmplY3Q+PHRleHQgeD0iNTAiIHk9IjE5IiBmaWxsPSIjRkZEOTY2IiBmb250LWZhbWlseT0iSGVsdmV0aWNhIiBmb250LXNpemU9IjEzIiBmb250LXdlaWdodD0iYm9sZCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+VGVjaCBQcmV2aWV3PC90ZXh0Pjwvc3dpdGNoPjxzd2l0Y2g+PGcgcmVxdWlyZWRGZWF0dXJlcz0iaHR0cDovL3d3dy53My5vcmcvVFIvU1ZHMTEvZmVhdHVyZSNFeHRlbnNpYmlsaXR5Ii8+PGEgeGxpbms6aHJlZj0iaHR0cHM6Ly93d3cuZGlhZ3JhbXMubmV0L2RvYy9mYXEvc3ZnLWV4cG9ydC10ZXh0LXByb2JsZW1zIiB0YXJnZXQ9Il9ibGFuayIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMCAtNSkiPjx0ZXh0IHg9IjUwJSIgeT0iMTAwJSIgZm9udC1zaXplPSIxMCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+VGV4dCBpcyBub3QgU1ZHIC0gY2Fubm90IGRpc3BsYXk8L3RleHQ+PC9hPjwvc3dpdGNoPjwvc3ZnPg==) Calico Cloud introduces malware detection in tech preview, which uses eBPF-based monitoring to log observed file hashes of programs running in your Calico Cloud Kubernetes clusters. Malware detection identifies malicious files by comparing observed file hashes with our threat intelligence library of known malware, and generates alerts when malware is detected in your cluster. Alerts can be viewed on the Alerts page of the web console. **If you started using Calico Cloud before January 24, 2022**, you must upgrade your existing cluster to get malware detection: 1. Navigate to the **Managed Clusters** page. 2. Select the cluster from the list, and click **Reinstall**. 3. Copy the updated install script command and run it against your cluster. ## April 20, 2022[​](#april-20-2022) ### Improved installation[​](#improved-installation) We’ve updated the Calico Cloud installation process to improve security, reduce dependencies on utilities (such as bash), and allow you to customize the name of your connected clusters. The Calico Cloud installation process will now require running a `kubectl apply` command instead of a bash script. Additionally, the installation script has been moved behind an authenticated endpoint. The updated install script is now available on the **Managed Clusters** page of the Calico Cloud UI. **If you started using Calico Cloud before January 24, 2022**, you must upgrade your existing cluster to get these changes: 1. Navigate to the **Managed Clusters** page. 2. Select the cluster from the list, and click **Reinstall**. 3. Copy the updated install script command and run it against your cluster. ## April 19, 2022[​](#april-19-2022) ### New feature: Image Assurance[​](#new-feature-image-assurance) ![tech-preview](data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnhsaW5rPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rIiB3aWR0aD0iMTAzIiBoZWlnaHQ9IjMzIiBzdHlsZT0iYmFja2dyb3VuZC1jb2xvcjojZmZmIiB2aWV3Qm94PSItMC41IC0wLjUgMTAzIDMzIj48ZGVmcz48c3R5bGU+QGltcG9ydCB1cmwoaHR0cHM6Ly9mb250cy5nb29nbGVhcGlzLmNvbS9jc3M/ZmFtaWx5PUJhcmxvdyk7PC9zdHlsZT48L2RlZnM+PHBhdGggZD0iTS0uMzQtLjMzSDEwMVYzMUgtLjM0eiIgcG9pbnRlci1ldmVudHM9ImFsbCIvPjxzd2l0Y2ggdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLS41IC0uNSkiPjxmb3JlaWduT2JqZWN0IHdpZHRoPSIxMDAlIiBoZWlnaHQ9IjEwMCUiIHBvaW50ZXItZXZlbnRzPSJub25lIiByZXF1aXJlZEZlYXR1cmVzPSJodHRwOi8vd3d3LnczLm9yZy9UUi9TVkcxMS9mZWF0dXJlI0V4dGVuc2liaWxpdHkiIHN0eWxlPSJvdmVyZmxvdzp2aXNpYmxlO3RleHQtYWxpZ246bGVmdCI+PGRpdiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94aHRtbCIgc3R5bGU9ImRpc3BsYXk6ZmxleDthbGlnbi1pdGVtczp1bnNhZmUgY2VudGVyO2p1c3RpZnktY29udGVudDp1bnNhZmUgY2VudGVyO3dpZHRoOjk5cHg7aGVpZ2h0OjFweDtwYWRkaW5nLXRvcDoxNXB4O21hcmdpbi1sZWZ0OjFweCI+PGRpdiBkYXRhLWRyYXdpby1jb2xvcnM9ImNvbG9yOiAjRkZEOTY2OyIgc3R5bGU9ImJveC1zaXppbmc6Ym9yZGVyLWJveDtmb250LXNpemU6MDt0ZXh0LWFsaWduOmNlbnRlciI+PGRpdiBzdHlsZT0iZGlzcGxheTppbmxpbmUtYmxvY2s7Zm9udC1zaXplOjEzcHg7Zm9udC1mYW1pbHk6SGVsdmV0aWNhO2NvbG9yOiNmZmQ5NjY7bGluZS1oZWlnaHQ6MS4yO3BvaW50ZXItZXZlbnRzOmFsbDtmb250LXdlaWdodDo3MDA7d2hpdGUtc3BhY2U6bm9ybWFsO292ZXJmbG93LXdyYXA6bm9ybWFsIj5UZWNoIFByZXZpZXc8L2Rpdj48L2Rpdj48L2Rpdj48L2ZvcmVpZ25PYmplY3Q+PHRleHQgeD0iNTAiIHk9IjE5IiBmaWxsPSIjRkZEOTY2IiBmb250LWZhbWlseT0iSGVsdmV0aWNhIiBmb250LXNpemU9IjEzIiBmb250LXdlaWdodD0iYm9sZCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+VGVjaCBQcmV2aWV3PC90ZXh0Pjwvc3dpdGNoPjxzd2l0Y2g+PGcgcmVxdWlyZWRGZWF0dXJlcz0iaHR0cDovL3d3dy53My5vcmcvVFIvU1ZHMTEvZmVhdHVyZSNFeHRlbnNpYmlsaXR5Ii8+PGEgeGxpbms6aHJlZj0iaHR0cHM6Ly93d3cuZGlhZ3JhbXMubmV0L2RvYy9mYXEvc3ZnLWV4cG9ydC10ZXh0LXByb2JsZW1zIiB0YXJnZXQ9Il9ibGFuayIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMCAtNSkiPjx0ZXh0IHg9IjUwJSIgeT0iMTAwJSIgZm9udC1zaXplPSIxMCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+VGV4dCBpcyBub3QgU1ZHIC0gY2Fubm90IGRpc3BsYXk8L3RleHQ+PC9hPjwvc3dpdGNoPjwvc3ZnPg==) Calico Cloud introduces Image Assurance in tech preview, enabling DevOps and platform teams to scan images in public and private registries, and images that are automatically discovered in connected clusters. Image Assurance provides a runtime view into risk, based on discovered vulnerabilities. It also offers admission controller policies to enforce how vulnerable images are used to create resources within Kubernetes. To get started, see [Image Assurance](https://docs.tigera.io/calico-cloud/image-assurance).