Gitpod Self-hosted installation on Tencent Cloud

    2022-06-10

    This post documents the process of self-hosting Gitpod (with a Community plan). There is also a Professional plan offering available at https://www.gitpod.io/self-hosted. Please check it out.

    One of the reasons I tried self-hosting Gitpod is to better understand its workspace orchestration & provision capability, which is hard to have a thorough understanding of, by just observing the user-space behavior. The widespread adoption of the Cloud is changing the Engineering Productivity landscape (which I’ve been working on), Gitpod demonstrates a path to a cloud-based development experience, which is fascinating to me. Also I need to setup a testbed to finish working on some PR for Gitpod’s OSS repository. So that’s how it began.

    The underlying host machine I was using is a Tencent Cloud Virtual Machine (CVM) with the SA2.2XLARGE16 specification, which is a standard model with balanced performance. The VM is an 8-core 16GB instance, and it supports a pay-as-you-go billing model. Note that Gitpod has a requirement on the Node’s Kernel version (≥ 5.4.0), so better choose Ubuntu Server 20.04 LTS for OS Image.

    Installation process

    1. Cluster setup

    To setup a Kubernetes cluster:

    • Install a single node k3s cluster (with custom node labels) following the k3s cluster setup guide.

      1.   $ export INSTALL_K3S_EXEC="server --disable traefik --flannel-backend=none \
      2.       --node-label gitpod.io/workload_meta=true \
      3.       --node-label gitpod.io/workload_ide=true \
      4.       --node-label gitpod.io/workload_workspace_regular=true \
      5.       --node-label gitpod.io/workload_workspace_headless=true \
      6.       --node-label gitpod.io/workload_workspace_services=true"
      7.   $ curl -sfL https://get.k3s.io | sh -
      8.   $ k3s -version
      9.   k3s version v1.23.6+k3s1 (418c3fa8)
      10.   go version go1.17.5

    • k3s installer generates its own kubeconfig file, so export the KUBECONFIG envvar for other tools like helm to access the k3s cluster. Note that k3s comes with kubectl baked in, so setting an alias to k3s kubectl is handier.

      1.   $ export KUBECONFIG=/etc/rancher/k3s/k3s.yaml # export KUBECONFIG=... && helm install
      2.   $ alias k="k3s kubectl"

    • Install the Calico Operator & CRD, and modify the calico-config based on the Gitpod installation guide.

      1.   $ k create -f https://projectcalico.docs.tigera.io/manifests/tigera-operator.yaml
      2.   $ k create -f https://projectcalico.docs.tigera.io/manifests/custom-resources.yaml
      4.   # download the Calico manifest <https://docs.projectcalico.org/manifests/calico-vxlan.yaml>
      5.   # add 1 line `"container_settings": { "allow_ip_forwarding": true }` to the `plugin` section
      6.   # copy the edited file to `/var/lib/rancher/k3s/server/manifests/`

    • Check the Node status again.

      1.   $ k get node -owide
      2.   NAME            STATUS   ROLES                  AGE   VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
      3.   vm-0-9-ubuntu   Ready    control-plane,master   1h    v1.23.6+k3s1   172.22.0.9    <none>        Ubuntu 20.04.4 LTS   5.4.0-109-generic   containerd://1.5.11-k3s2

    2. Install Cert-Manager & Configure DNS

    The Certificate and Networking managements are well-known hard problems for software developers, even without the complexity of Kubernetes. I tried several approaches to avoid this particular rabbit hole, and later found out it’s better just to learn the necessary prior knowledge and follow the installation guide.

    Some basic understanding I got:

    • cert-manager is the “Certificates as a Service” in the Kubernetes ecosystem. It introduces CRDs like Certificate / Issuer into the Kubernetes API.

    • ACME (Automated Certificate Management Environment) is a protocol proposed by Let’s Encrypt. It allows an ACME client (e.g. cert-manager) to request (also renew/revoke) a certificate automatically from CA (e.g. Let’s Encrypt)
    • DNS-01 challenge is a domain validation procedure, as you prove to CA that you control the domain name by putting a specific TXT record under it (by making API calls to the DNS provider). Different from other challenge types (such as HTTP-01), DNS-01 allows issuing wildcard certificates (which Gitpod requires).

    Move on to the installation:

    • Configure root domain (@) and two wildcard subdomains (* and *.ws) from the DNS provider, pointing to the public IP of the VM. Make some digs to check if the settings work.

      1.   $ dig <mygitpod.domain>
      2.   A <mygitpod.domain>. 9m11s   43.156.xx.xx
      3.   $ dig test.<mygitpod.domain>
      4.   A test.<mygitpod.domain>. 9m41s   43.156.xx.xx
      6.   $ which dig
      7.   dig: aliased to dog

    • Install cert-manager.

      1.   $ k apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml
      2.   $ k get pod -A
      3.   NAMESPACE         NAME                                       READY   STATUS    RESTARTS   AGE
      4.   kube-system       calico-node-mmff6                          1/1     Running   0          18m
      5.   kube-system       coredns-d76bd69b-9glsl                     1/1     Running   0          21m
      6.   kube-system       calico-kube-controllers-6b77fff45-dt8lg    1/1     Running   0          18m
      7.   kube-system       metrics-server-7cd5fcb6b7-r8wsj            1/1     Running   0          21m
      8.   kube-system       local-path-provisioner-6c79684f77-vw9js    1/1     Running   0          21m
      9.   tigera-operator   tigera-operator-7d8c9d4f67-9n5c8           1/1     Running   0          17m
      10.   cert-manager      cert-manager-64d9bc8b74-rgng9              1/1     Running   0          20s
      11.   cert-manager      cert-manager-cainjector-6db6b64d5f-gvgwx   1/1     Running   0          20s
      12.   cert-manager      cert-manager-webhook-6c9dd55dc8-7gzbt      1/1     Running   0          20s

    • Create API Token from the DNS provider console (I was using DNSPod).

    cert-manager does not have built-in support for DNSPod, however it supports Webhook resolver for these out-of-tree DNS providers. I tried cert-manager-webhook-dnspod and it worked out great. It also has well-documented instructions on the Tencent Cloud doc site.

    • Install cert-manager-webhook-dnspod using Helm with custom options.

      1.   $ git clone --depth 1 https://github.com/qqshfox/cert-manager-webhook-dnspod.git
      2.   $ helm install --name cert-manager-webhook-dnspod ./deploy/cert-manager-webhook-dnspod \
      3.       --namespace cert-manager \
      4.       --set groupName=<GROUP_NAME> \
      5.       --set secrets.apiID=<DNSPOD_API_ID>,secrets.apiToken=<DNSPOD_API_TOKEN> \
      6.       --set clusterIssuer.enabled=true,clusterIssuer.email=<EMAIL_ADDRESS>

    • Check the generated ClusterIssuer. If the “READY” state keeps False, describe the ClusterIssuer to see if there is an exception.

      1.   $ k get ClusterIssuer -A
      2.   NAME                                         READY   AGE
      3.   cert-manager-webhook-dnspod-cluster-issuer   True    1h
      5.   $ k describe ClusterIssuer cert-manager-webhook-dnspod-cluster-issuer -n cert-manager
      6.   ...
      7.   Spec:
      8.     Ca:
      9.       Secret Name:  cert-manager-webhook-dnspod-ca
      10.   Status:
      11.     Conditions:
      12.       Last Transition Time:  2022-06-07T11:19:16Z
      13.       Message:               Signing CA verified
      14.       Observed Generation:   1
      15.       Reason:                KeyPairVerified
      16.       Status:                True
      17.       Type:                  Ready
      18.   Events:
      19.     Type    Reason           Age                From                  Message
      20.     ----    ------           ----               ----                  -------
      21.     Normal  KeyPairVerified  16m (x2 over 16m)  cert-manager-issuers  Signing CA verified

    • Create a Certificate.

      1.   # cert.yaml
      2.   apiVersion: cert-manager.io/v1
      3.   kind: Certificate
      4.   metadata:
      5.     name: my-crt
      6.     namespace: default # choose your namespace
      7.   spec:
      8.     secretName: my-crt-secret
      9.     issuerRef:
      10.       name: cert-manager-webhook-dnspod-cluster-issuer # refs to the ClusterIssuer
      11.       kind: ClusterIssuer
      12.       group: cert-manager.io
      13.     dnsNames: # your dnsNames
      14.     - "mygitpod.domain"
      15.     - "*.mygitpod.domain"
      16.     - "*.ws.mygitpod.domain"

    • Wait for the certificate’s “READY” state to become True (could be minutes).

      1.   $ k apply -f cert.yaml
      2.   $ k get cert -A
      3.   NAMESPACE      NAME                                      READY   SECRET                                    AGE
      4.   cert-manager   cert-manager-webhook-dnspod-ca            True    cert-manager-webhook-dnspod-ca            1h
      5.   cert-manager   cert-manager-webhook-dnspod-webhook-tls   True    cert-manager-webhook-dnspod-webhook-tls   1h
      6.   cert-manager   my-crt                                    True    my-crt-secret                             1h

    • Check and validate the certificate (optional). tls.key / tls.crt can be found in secret my-crt-secret

      1.   $ k describe cert my-crt -n cert-manager
      2.   ...
      3.   Spec:
      4.     Dns Names:
      5.       mygitpod.domain
      6.       *.mygitpod.domain
      7.       *.ws.mygitpod.domain
      8.     Issuer Ref:
      9.       Group:      cert-manager.io
      10.       Kind:       ClusterIssuer
      11.       Name:       cert-manager-webhook-dnspod-cluster-issuer
      12.     Secret Name:  my-crt-secret
      13.   Status:
      14.     Conditions:
      15.       Last Transition Time:  2022-06-07T11:28:39Z
      16.       Message:               Certificate is up to date and has not expired
      17.       Observed Generation:   1
      18.       Reason:                Ready
      19.       Status:                True
      20.       Type:                  Ready
      21.     Not After:               2022-09-05T10:28:38Z
      22.     Not Before:              2022-06-07T10:28:39Z
      23.     Renewal Time:            2022-08-06T10:28:38Z
      24.     Revision:                1
      25.   Events:                    <none>
      27.   # dump certificates if you need
      28.   $ k get secret mytest1111-cloud-crt-secret -n cert-manager -o jsonpath='{.data.tls\.crt}' | base64 -d

    3. Install Gitpod

    Gitpod previously used Helm for self-hosted installation, and recently it switched from Helm to a custom installer. There is a blog explaining some technical reasons: https://www.gitpod.io/blog/gitpod-installer. Gitpod now suggests using kots, which might be based on installer as well, but with a pretty WebUI console.

    • Install kots plugin & install Gitpod using kots.

      1.   $ curl https://kots.io/install | bash
      2.   $ k kots install gitpod
      3.   Enter the namespace to deploy to: gitpod
      4.     • Deploying Admin Console
      5.       • Creating namespace ✓
      6.       • Waiting for datastore to be ready ✓
      7.   Enter a new password to be used for the Admin Console: •••••••••
      8.     • Waiting for Admin Console to be ready ✓
      10.     • Press Ctrl+C to exit
      11.     • Go to http://localhost:8800 to access the Admin Console

    • Admin-console listened on localhost:8800 only, so setup an Nginx server to proxy the requests.

      1.   $ apt install nginx-full
      2.   $ vi /etc/nginx/nginx.conf
      4.   # some piece of nginx.conf
      5.   ...
      6.   http {
      7.       server {
      8.           listen 8800;
      9.           location / {
      10.               proxy_pass  http://127.0.0.1:8800/;
      11.           }
      12.       }
      13.       # include /etc/nginx/conf.d/*.conf;
      14.       # include /etc/nginx/sites-enabled/*;
      15.       ...
      16.   }
      17.   ...
      19.   $ nginx

    Now visit the admin-console from your browser, the only notable part is the “Issuer name” of TLS certificates settings (cert-manager-webhook-dnspod-cluster-issuer). Hit “Continue” to run the preflight check and wish it all green.

    • Once the installation is finished, check if all the workloads are ready.

      1.   $ k get pod -n gitpod
      2.   NAME                                  READY   STATUS
      3.   kotsadm-minio-0                       1/1     Running
      4.   kotsadm-postgres-0                    1/1     Running
      5.   kotsadm-bb4b8b869-d879q               1/1     Running
      6.   gitpod-telemetry-27576960-n95md       0/1     Completed
      7.   installation-status-67d64d7cd-88wk7   1/1     Running
      8.   svclb-proxy-9r8ws                     3/3     Running
      9.   registry-776df46bd6-rw4jk             1/1     Running
      10.   dashboard-7bdd74f889-wflkr            1/1     Running
      11.   agent-smith-h44fq                     2/2     Running
      12.   openvsx-proxy-0                       2/2     Running
      13.   image-builder-mk3-7c7fb9fddb-x9rrs    2/2     Running
      14.   ws-manager-76cb58c9fd-49zfv           2/2     Running
      15.   content-service-db499c5c4-6zlwc       1/1     Running
      16.   ide-proxy-56dc5f7d44-xqldw            1/1     Running
      17.   minio-5c684d5449-lsk28                1/1     Running
      18.   ws-daemon-sbb5f                       2/2     Running
      19.   blobserve-586448694c-jwnwq            2/2     Running
      20.   ws-proxy-64f67cd7b5-tn6nn             2/2     Running
      21.   mysql-0                               1/1     Running
      22.   messagebus-0                          1/1     Running
      23.   proxy-6fd7c89748-bl8ts                2/2     Running
      24.   registry-facade-dlzrq                 2/2     Running
      25.   server-845864bb59-vfzwv               2/2     Running
      26.   ws-manager-bridge-86c78c4487-ssbqv    2/2     Running

    4. Start a Workspace

    Now visit the domain name and the Gitpod dashboard should be alive. Configure code host integration (such as GitHub) the same way you do with Gitpod SaaS.

    Some notable points when creating a workspace:

    • When bootstrapping the first workspace, Gitpod would pull a 3GiB+ workspace-full image. If you have limited bandwidth or an unstable network connection, this process could be painful.
    • Gitpod tries to communicate with the in-cluster registry https://registry.mygitpod.domain/v2/workspace-image for committing additional layers, however the registry host resolves to the public IP, which causes a “hairpin” — travel out of the cluster and then back in via the external IP, thus got lots of timeout exception.

    @iQQBot advised to configure kubedns to add a rewrite rule mapping registry.mygitpod.domain to proxy.gitpod.svc.cluster.local, and then the registry host would resolve to the internal ClusterIP. proxy is the component for TLS termination and traffic proxy. Note that just setting the registry host URL to registry service FQDN from gitpod configMap wouldn’t work (cert validation would fail).

    After the modification, the workspace pod turned to Running state quickly and finally web browser proceeded to the workspace view.

    Conclusions

    The official doc is pretty high quality, I did have several failed attempts when trying to bypass or violate some requirements, and got all kinds of error messages, from containerd failures (yield by ws-daemon when using a CentOS-based host), certificate failures (crictl pull & other system tools failed when using a self-signed cert first, and later stuck at issuing ACME certs), network connectivity issues, etc. I spent the first several nights just fighting these weird problems and got no progress at all, when I started over just followed all the requirements from Gitpod’s guide, and it worked out quickly.

    A basic familiarity with Kubernetes and the Gitpod architecture is necessary. Without the help from Gitpod member @iQQBot (Thanks again @iQQBot!), I would possibly be blocked somewhere and just given up during the process.

    Though this single node cluster is far from a production-ready development infrastructure, and there are lots of black boxes across the whole system, it did give me a chance to see how Gitpod build the magic product from an inside view.

    Some suggestions on Cloud-hosting service selection:

    • Put more resources onto the disk & network part (e.g. choose a High I/O CVM instance model), especially if you’re using the in-cluster registry & object storage.

    Thank you so much for reading. This post is also a part of a remote development series:

    1. Gitpod workspace with JetBrains Gateway
    2. Gitpod Self-hosted installation on Tencent Cloud
    3. How JetBrains Gateway works - beyond the black box