文件列表:

.sops.yaml (335, 2023-04-21)
LICENSE (16725, 2023-04-21)
bootstrap.env (3848, 2023-04-21)
cluster (0, 2023-04-21)
cluster\apps (0, 2023-04-21)
cluster\apps\code-server (0, 2023-04-21)
cluster\apps\code-server\config-pvc.yaml (211, 2023-04-21)
cluster\apps\code-server\helm-release.yaml (1353, 2023-04-21)
cluster\apps\code-server\kustomization.yaml (116, 2023-04-21)
cluster\apps\esphome (0, 2023-04-21)
cluster\apps\esphome\config-pvc.yaml (210, 2023-04-21)
cluster\apps\esphome\helm-release.yaml (1259, 2023-04-21)
cluster\apps\esphome\kustomization.yaml (116, 2023-04-21)
cluster\apps\home-assistant (0, 2023-04-21)
cluster\apps\home-assistant\code-server.sops.yaml (5765, 2023-04-21)
cluster\apps\home-assistant\config-pvc.yaml (220, 2023-04-21)
cluster\apps\home-assistant\helm-release.yaml (2551, 2023-04-21)
cluster\apps\home-assistant\home-assistant.sops.yaml (1707, 2023-04-21)
cluster\apps\home-assistant\kustomization.yaml (171, 2023-04-21)
cluster\apps\influxdb (0, 2023-04-21)
cluster\apps\influxdb\config-pvc.yaml (215, 2023-04-21)
cluster\apps\influxdb\helm-release.yaml (1909, 2023-04-21)
cluster\apps\influxdb\kustomization.yaml (117, 2023-04-21)
cluster\apps\kube-dashboard (0, 2023-04-21)
cluster\apps\kube-dashboard\helm-release.yaml (1122, 2023-04-21)
cluster\apps\kube-dashboard\kustomization.yaml (96, 2023-04-21)
cluster\apps\kustomization.yaml (173, 2023-04-21)
cluster\apps\qbittorrent (0, 2023-04-21)
... ...

# Overview Deploy a local Kubernetes datacenter that features low maintenance, high security, and simple scalability. To achieve these goals, the following methodologies will be utilized: - [GitOps](https://www.weave.works/technologies/gitops/) with [Flux](https://fluxcd.io/docs/concepts/) & [Terraform Cloud](https://cloud.hashicorp.com/products/terraform) - Bare metal Kubernetes deployment with [k3os](https://k3os.io/) - Multi-layer encryption using [SOPs](https://fluxcd.io/docs/guides/mozilla-sops/) & [Cert Manager](https://cert-manager.io/docs/) - [Zero Trust security](https://www.cloudflare.com/learning/security/glossary/what-is-zero-trust/) with [Cloudflare](https://www.cloudflare.com/what-is-cloudflare/) & [2FA](https://docs.github.com/en/authentication/securing-your-account-with-two-factor-authentication-2fa/configuring-two-factor-authentication) - Automated system updates with [Renovate](https://www.whitesourcesoftware.com/free-developer-tools/renovate/) & [System Upgrade Controller](https://rancher.com/docs/k3s/latest/en/upgrades/automated/) # Diagrams      # Prerequisites In order to complete this guide, you will need the following: - MacOS or Linux - Visual Studio Code - [Homebrew](https://brew.sh/) - A GitHub Account - A Cloudflare Account with a domain - A Terraform Cloud Account - A PC/VM with at least 8GB RAM - A positive attitude and some patience # Deployment overview This guide will walk you through the following steps: 1. Setup 1. Fork this repo 1. OS Installation 1. Connect to your Kubernetes node 1. Generate a Cloudflare API key 1. Activate Cloudflare Zero Trust 1. Generate a Terraform Cloud API token 1. Generate a GitHub OAuth token for Cloudflare 1. Generate a GitHub Personal Access Token for Terraform Cloud 1. Configure secrets encryption 1. Prepare for deployment 1. Deployment 1. Configure and Deploy Flux 1. Deploy your Kubernetes cluster resources 1. Automate external resource creation with Terraform Cloud 1. Access your apps from anywhere 1. Security 1. Extend Zero Trust security 1. Threat protection and visibility with DNS layer security 1. Additional Automation 1. Automate your app updates with Renovate Bot 1. Automate k3s updates 1. Operations 1. Add your own apps 1. Observability, health-checking, and performance 1. Extras 1. Visualize your repo **Note**: In order to stay focused on secure GitOps practices, these practices will not be covered in depth within this guide: - High availability for networking, storage, and Kubernetes components - Hosted/Cloud deployments - Load balancing - Automated secrets management - Disaster recovery ## Fork this repo In addition to this repository containing all the resources you will use throughout this guide, your GitHub repository will be the single source of truth for your Kubernetes & Cloudflare infrastructure definitions. When new code is merged into your GitHub repository, Flux (which we will setup in a later step) will ensure your environment reflects the state of your GitHub repository. This is the essense of Infrastructure as code (IaC) - the practice of keeping all your declarative infrastructure configuration stored as code. 1. [Fork](https://docs.github.com/en/get-started/quickstart/fork-a-repo) this `k3s-gitops` repo into your own GitHub repo. ## OS Installation k3OS is a stripped-down, streamlined, easy-to-maintain operating system for running Kubernetes nodes. 1. Download the respective ISO (within the latest release assets) in the [k3OS repo](https://github.com/rancher/k3os/releases). 1. For dedicated PC users: 1. Create a bootable USB stick from the ISO with [balenaEtcher](https://www.balena.io/etcher/). 1. For VM users: 1. Mount the ISO to your VM. 1. Boot to the ISO on your respective device. 1. Select the "k3OS LiveCD and Installer" option on the boot menu. 1. Login with `rancher/rancher` 1. Run `lsblk` to identify your desired destination drive for k3OS. ```sh lsblk #NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT #loop1 7:1 0 47.1M 1 loop /usr #loop2 7:2 0 302.2M 0 loop /usr/src #sda 8:0 0 238.5G 0 disk #├─sda1 8:1 0 47M 0 part #└─sda2 8:2 0 238.4G 0 part /var/lib/kubelet/pods/2d4b1a97-6659-436d-abb1- #sdb 8:16 0 7.6G 0 disk #├─sdb1 8:17 0 190K 0 part #└─sdb2 8:18 0 2.8M 0 part #├─sdb3 8:19 0 513.5M 0 part /k3os/system #└─sdb4 8:20 0 300K 0 part ``` 1. Begin the installation process. ```sh sudo k3os install ``` 1. Choose to install to disk. ```log Running k3OS configuration Choose operation 1. Install to disk 2. Configure server or agent Select Number [1]: ``` 1. Choose the desired destination drive you identified earlier. ```log Installation target. Device will be formatted 1. sda 2. sdb Select Number [0]: 1 ``` 1. Choose NOT to configure the system with `cloud-init`. ```log Config system with cloud-init file? [y/N]: N ``` 1. Choose NOT to authorize GitHub users for SSH. ```log Authorize GitHub users to SSH [y/N]: N ``` 1. Choose to keep the default password for the `rancher` account (feel free to change it later). ```log Please enter password for [rancher]: Confirm password for [rancher]: ``` 1. Choose NOT to configure WiFi (WiFi does not support manual static IP assignment, which is necessary for a later step). ```log Configure WiFi? [y/N]: N ``` 1. Choose to run your node as a server. ```log Run as a server or agent? 1. server 2. agent Select Number [1]: ``` 1. Set your cluster secret/token as `cluster-secret` (this can be changed later as desired). ```log Token or cluster secret (optional): cluster-secret ``` 1. Confirm your configuration details and enter `y` to continue. ```log Configuration _____________ device: /dev/sda Your disk will be formatted and k3OS will be installed with the above configuration. Continue? [y/N]: y ``` **Note**: If you receive install errors using the LiveCD option, reboot and proceed with the same options using the "k3OS installer" option instead of the "k3OS LiveCD & Installer" option. 1. After the system completes the installation and reboots, select the `k3OS Current` bootloader option (or just wait a few moments and it will boot it by default). 1. Login to your new K3OS installation as `rancher` with the password `rancher` 1. Set a static IP on the nodes OS itself and NOT by using DHCP. Using DHCP to assign IPs injects a search domain into your nodes `/etc/resolv.conf` and this could potentially break DNS in containers. (If you already have a DHCP address assigned, remove any `search domain.com` lines from `/etc/resolv.conf` and save the file). 1. First, identify the `connman` service bound to `eth0` ```sh sudo connmanctl services # *AO Wired ethernet_84470907c635_cable ``` 1. View the details of your connection. ```sh sudo connmanctl services ethernet_84470907c635_cable ``` 1. Set a static IP address and DNS nameserver for your connection. ```sh sudo connmanctl config ethernet_84470907c635_cable --ipv4 manual 192.168.1.151 255.255.255.0 192.168.1.1 --nameservers 192.168.1.1 ``` 1. By default, k3OS allows SSH connections only using certificates. This is a much safer method than using passwords, however, for the sake of simplicity in this guide, set `PasswordAuthentication` to yes. Feel free to come back later and lock this down. 1. Open the SSHD configuration file ```sh sudo vim /etc/ssh/sshd_config ``` 1. Change the value of `PasswordAuthentication` from `no` to `yes` and save this file. 1. Restart the `sshd` service. ```sh sudo service sshd restart ``` 1. Log out of the console and grab your Mac. ```sh exit ``` You now have a k3OS server node ready for remote configuration. ## Connect to your Kubernetes node The majority of interaction with your Kubernetes node will occur from a remote development system - in this case, the Mac where you cloned this repo. 1. Connect to your new k3os node via SSH. ```log ssh rancher@192.168.1.183 The authenticity of host '192.168.1.183 (192.168.1.183)' can't be established. ECDSA key fingerprint is SHA256:/KPEdx6D56R9/ByhJr/4gGSP7DJdtkFun+fFgCtdl/Q. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '192.168.1.183' (ECDSA) to the list of known hosts. rancher@192.168.1.183's password: rancher Welcome to k3OS! ... ``` 1. In your `k3os-gitops` repo, copy the contents of `/k3os/node-config.yaml` to your clipboard. 1. On your k3os node, create a new file called `config.yaml` in the `/var/lib/rancher/k3os/` directory. ```sh sudo vim /var/lib/rancher/k3os/config.yaml ``` 1. Paste your clipboard contents in and save the file. 1. In k3OS, the `/etc` automatically reverts any changes after reboot. Therefore, to persistently change Hostname of k3OS machine, we have to change it in k3OS configuration files. 1. Edit the k3OS hostname file. ```sh sudo vim /var/lib/rancher/k3os/hostname ``` 1. Replace the current contents with your desired hostname and save the file. 1. Reboot your system for the system modules and hostname changes to take effect. ```sh sudo reboot ``` 1. On your Mac terminal, retrieve the kube config file from your k3OS node. ```sh scp rancher@192.168.1.151:/etc/rancher/k3s/k3s.yaml . #rancher@192.168.1.151's password: ``` 1. Open the `k3s.yaml` file you downloaded on your Mac. 1. Replace _127.0.0.1_ in the line with `server: https://127.0.0.1:***43` with the IP address of your Kubernetes node (for example, `server: https://192.168.1.151:***43`) 1. Copy the complete contents of `k3s.yaml` to your clipboard. 1. On your Mac, open (or create) the file `/Users/YOURUSERNAME/.kube/config` 1. Paste the contents of `k3s.yaml` into `/Users/YOURUSERNAME/.kube/config` and save the file. 1. Install `kubectl`. ```sh brew install kubectl ``` 1. Use `kubectl` to list your available Kubernetes nodes ```sh kubectl get nodes #NAME STATUS ROLES AGE VERSION #k3s-node2 Ready control-plane,master 23h v1.22.2+k3s2 ``` You are now able to securely access your active Kubernetes node from your remote development system. ## Generate a Cloudflare API key Cloudflare is used throughout this guide for several reasons: - Enables `cert-manager` to utilize the Cloudflare DNS challenge for automating TLS certificate creation in your Kubernetes cluster - Enables accessibility of your apps from anywhere - Secures access to your apps with Cloudflare Access - Provides DNS security, detailed traffic metrics, and logging with Cloudflare Gateway - Provides you with Zero Trust security capabilities - Provides you with an single-sign-on (SSO) portal for your apps - Integrates with Terraform Cloud for automated Cloudflare resource creation 1. Login to your [Cloudflare account](https://dash.cloudflare.com/login). 1. Create an API key by going to [this page](https://dash.cloudflare.com/profile/api-tokens) in your Cloudflare profile. **Note**: Your API key is a sensitive credential that allows programatic access to your Cloudflare account - ensure you take all precautions to protect this key. 1. Copy the API key to your clipboard. 1. Paste your API key as the value for `CLOUDFLARE_APIKEY` in your `bootstrap.env` file, then save the file. You now have a Cloudflare API key that will enable you to programatically create Cloudflare and encryption resources with ease. ## Activate Cloudflare Zero Trust Cloudflare Zero Trust is a free suite of Zero Trust security tools including Cloudflare Access and Cloudflare Gateway. In order to programatically utilize these features, you must first activate the service on your Cloudflare account and generate a team name attribute. https://developers.cloudflare.com/cloudflare-one/faq/teams-getting-started-faq 1. Visit the Cloudflare Zero Trust [sign up page](https://dash.cloudflare.com/sign-up/teams). 1. Follow the onboarding steps and choose a team name. 1. Copy your team name to your clipboard, paste your team name as the value for `CLOUDFLARE_TEAM_NAME` in your `bootstrap.env` file, then save the file. You now have the foundation for programatically integrating Cloudflare's Zero Trust tools into your environment. ## Generate a Terraform API token Terraform Cloud is an infrastructure-as-code tool that allows you to easily create external resources for Cloudflare and hundreds of other cloud services. Rather than manage a consistent state in each cloud service UI, Terraform allows you to define and manage these resources in your GitHub repository. This enables you to stay consistent with the philosophy of GitOps and streamline your CI/CD workflow. 1. Login to your [Terraform Cloud account](https://app.terraform.io/). 1. Create an API token by going to [this page](https://app.terraform.io/app/settings/tokens) in your Terraform Cloud profile. **Note**: Your API token is a sensitive credential that allows programatic access to your Terraform Cloud account - ensure you take all precautions to protect this key. 1. Copy the API token to your clipboard. 1. Paste your API token as the value for `TERRAFORM_CLOUD_TOKEN` in your `bootstrap.env` file, then save the file. You now have a Terraform Cloud API token that will enable you to programatically configure your Terraform environment. ## Generate a GitHub OAuth token for Cloudflare GitHub integrates with Cloudflare to secure your environment using Zero Trust security methodologies for authentication. Cloudflare will utilize your GitHub OAuth token to authorize user access to your applications. This will enable your GitHub identity to use Single Sign On (SSO) for all of your applications. 1. Login to your [GitHub account](https://github.com/login). 1. Go to the [OAuth token creation page](https://github.com/settings/developers), select "OAuth Apps", then click "Register a new application". **Note**: Your OAuth token is a sensitive credential - ensure you take all precautions to protect this key. 1. Complete the "Register a new OAuth application" form using these values. 1. Application name: `Cloudflare` 1. Homepage URL: `https://.cloudflareaccess.com` 1. Authorization callback URL: `https://.cloudflareaccess.com/cdn-cgi/access/callback` **Note**: Replace `` in the fields above with the contents of `CLOUDFLARE_TEAM_NAME` in your `bootstrap.env` file. 1. Click the "Register application" button once complete. 1. Copy the OAuth Client ID to your clipboard. 1. Paste your API token as the value for `CLOUDFLARE_OAUTH_CLIENT_ID` in your `bootstrap.env` file, then save the file. 1. Copy the OAuth Client Secret to your clipboard. 1. Paste your API token as the value for `CLOUDFLARE_OAUTH_CLIENT_SECRET` in your `bootstrap.env` file, then save the file. You now have a GitHub OAuth client and secret that will enable you to programatically configure your Cloudflare environment with Zero Trust security methodologies. ## Generate a GitHub Personal Access Token for Terraform Cloud A GitHub Personal Access Token (PAT) enables the integration between Terraform Cloud and your GitHub repository. This further enables the GitOps model by allowing Terraform Cloud to automatically initiate Terraform runs when changes are committed to your GitHub repository. 1. Follow [this GitHub guide](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) to create a Personal Access Token (PAT) with the following permission scopes: - repo:status - public_repo - read:repo_hook 1. Copy the GitHub Personal Access token to your clipboard. 1. Paste your API token as the value for `GITHUB_PERSONAL_ACCESS_TOKEN` in your `bootstrap.env` file, then save the file. You now have a GitHub Personal Access Token (PAT) that will enable you to programatically integrate your Terraform Cloud and GitHub instances. ## Configure Secrets Encryption Secrets encryption allows you to safely store secrets in a public or private Git repository. To accomplish this, [Age](https://github.com/FiloSottile/age) is a tool that will encrypt your YAML files and/or secrets using Mozilla SOPs (Secrets Operations) encryption. In a later step, you will configure Flux with this SOPs encryption key - this will allow your Kubernetes cluster to decrypt and utilize those secrets for operations. 1. Begin by installing `age`. ```sh brew install age ``` 1. Create a Age Private / Public Key. ```sh age-keygen -o age.agekey ``` 1. Set up the directory for the Age key and move the Age file to it. ```sh mkdir -p ~/.config/sops/age mv age.agekey ~/.config/sops/age/keys.txt ``` 1. Fill out the `age` public key in the `bootstrap.env` under `AGE_PUBLIC_KEY`. **Note**: The public key should start with `age`... 1. For disaster recovery purposes, copy the contents of your age private key to your (hopefully MFA-protected) password manager such as 1Password or LastPass. Your environment is now prepared for encrypting all secrets in your cluster. ## Prepare for deployment To prepare for deployment, it's necessary to bootstrap your development environment with your custom values such as DNS information, API keys, and encryption secrets. You'll then encrypt all your sensitive values before pushing your project to your Github repository. It is important to follow these steps carefully to ensure no sensitive values are pushed to your public repository. 1. Open and edit your `bootstrap.env` file to ensure it includes all your respective unique values, then save the file. **Note**: Some variables contain the prefix `TF_VAR_` - This prefix enables Terraform to use your local environment variables for Terraform runs. 1. Source the `bootstrap.env` file to set the respective environment variables in your terminal. ```sh source bootstrap.env ``` 1. In the same terminal window where you set your environment variables, run the following commands to create your unique, encrypted deployment files. ```sh # create sops configuration file envsubst < "${PROJECT_DIR}/tmpl/.sops.yaml" \ > "${PROJECT_DIR}/.sops.yaml" # create unique cluster resources envsubst < "${PROJECT_DIR}/tmpl/cluster/cluster-settings.yaml" \ > "${PROJECT_DIR}/cluster/base/cluster-settings.yaml" envsubst < "${PROJECT_DIR}/tmpl/cluster/gotk-sync.yaml" \ > "${PROJECT_DIR}/cluster/base/flux-system/gotk-sync.yaml" envsubst < "${PROJECT_DIR}/tmpl/cluster/cluster-secrets.sops.yaml" \ > "${PROJECT_DIR}/cluster/base/cluster-secrets.sops.yaml" envsubst < "${PROJECT_DIR}/tmpl/cluster/cert-manager-secret.sops.yaml" \ > "${PROJECT_DIR}/cluster/core/cert-manager/secret.sops.yaml" # encrypt sensitive files sops --encrypt --in-place "${PROJECT_DIR}/cluster/base/cluster-secrets.sops.yaml" sops --encrypt --in-place "${PROJECT_DIR}/cluster/core/cert-manager/secret.sops.yaml" ``` **Note**: Variables ... ...

近期下载者：

相关文件：

评论：[我要评论] [举报此文件]

收藏者：