Forgejo instances running in K8S

Deployments

The webhooks of this repository will update existing deployments by sending a POST request. For instance https://v200.next.forgejo.org/.well-known/wakeup-on-logs/forgejo-v200 will upgrade v200.next.forgejo.org.

Current

• https://v9.next.forgejo.org
• https://v8.next.forgejo.org
• https://v7.next.forgejo.org (referenced here for completness but not deployed using Helm, it updated hourly using an ad-hoc shell script)

Disabled

They are online because they contain information referenced from the Forgejo issue tracker to demonstrate a problem. But they can no longer be used for testing because they are either obsolete or running a Forgejo instance that is EOL.

       -e FORGEJO__service__DISABLE_REGISTRATION=true \
       -e FORGEJO__actions__ENABLED=false \
       -e FORGEJO__mirror__ENABLED=false \

• https://next.forgejo.org

Offline

These instances are offline but archived and can be booted for forensice analysis if neeeded.

LXC container

version=9
name=forgejo-v$version

lxc-helpers.sh lxc_container_create --config "k8s" $name
echo "lxc.start.auto = 1" | sudo tee -a /var/lib/lxc/$name/config
lxc-helpers.sh lxc_container_start $name
lxc-helpers.sh lxc_container_user_install $name $(id -u) $USER

K3S installation

lxc-helpers.sh lxc_container_run $name -- sudo --user debian bash
echo 'export TERM=xterm-256color' >> .bashrc
echo 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml' >> .bashrc
exit

lxc-helpers.sh lxc_container_run $name -- sudo --user debian bash
sudo apt-get install curl
curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE=0644 sh -
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash -

Setup the domain

• In the registrar v$version.next.forgejo.org CNAME hetzner04.forgejo.org.

ip=$(lxc-helpers.sh lxc_container_run $name -- hostname -I | cut -f1 -d' ')
sudo tee /etc/nginx/sites-available/v$version.next.forgejo.org <<'EOF'
server {
    listen 80;
    listen [::]:80;

    server_name v{VERSION}.next.forgejo.org;

    location / {
        proxy_pass http://{IP}:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        client_max_body_size 2G;
    }
}
EOF
sudo sed -i -e "s/{IP}/$ip/" -e "s/{VERSION}/$version/" /etc/nginx/sites-available/v$version.next.forgejo.org

( cd /etc/nginx/sites-enabled ; ln -s /etc/nginx/sites-available/v$version.next.forgejo.org )
sudo certbot -n --agree-tos --email contact@forgejo.org -d v$version.next.forgejo.org --nginx

Forward SSH

port=$(printf "2%02d0" $version)
cat > /home/debian/v$version.nftables <<EOF
add table ip v$version;
flush table ip v$version;
add chain ip v$version prerouting {
  type nat hook prerouting priority 0;
  policy accept;
  dnat ip addr . port to tcp dport map { $port : $ip . 2222 };
};
EOF

• Add to iface enp4s0 inet static in /etc/network/interfaces

  up nft -f /home/debian/v$version.nftables
  

Define the wakeup-on-logs script

cd /etc/wakeup-on-logs
sudo ln -s forgejo-v8 forgejo-v9
sudo systemctl restart wakeup-on-logs-watch

#!/bin/bash

set -x

self="${BASH_SOURCE[0]}"
name=$(basename $self)
# keep it lower than https://code.forgejo.org/infrastructure/wakeup-on-logs
# otherwise it will get killed by it
timeout=4m

function lxc_run() {
    lxc-attach $name -- sudo --user debian KUBECONFIG=/etc/rancher/k3s/k3s.yaml "$@" |& tee -a /var/log/$name.log
}

image=codeberg.org/forgejo-experimental/forgejo
major=${name##*v}
digest=$(skopeo inspect --format "{{.Digest}}" docker://$image:$major-rootless)
values=https://code.forgejo.org/infrastructure/k8s/raw/branch/main/forgejo-v$major/values.yml
lxc_run helm upgrade forgejo -f $values -f /home/debian/secrets.yml oci://code.forgejo.org/forgejo-helm/forgejo --atomic --wait --timeout $timeout --install --set image.digest=$digest

Define Forgejo Helm values

• https://code.forgejo.org/infrastructure/k8s/src/branch/main/forgejo-v$version/values.yml
• Add https://v$version.next.forgejo.org/.well-known/wakeup-on-logs/forgejo-v$version as a webhook https://code.forgejo.org/infrastructure/k8s/settings/hooks

Note that it requires the $version.0-test release to be published before it can successfully deploy. Otherwise it will timeout because the pod fails:

debian@forgejo-v9:~$ kubectl get pods
NAME                      READY   STATUS              RESTARTS   AGE
forgejo-ff4fb8767-hhdnk   0/1     Init:ErrImagePull   0          3m25s

Define Forgejo Helm secrets

cat secrets.yml
gitea:
  admin:
    password: "XXX"
  config:
    mailer:
      PASSWD: "YYY"
( cd /var/lib/lxc ; cp -a forgejo-v8/rootfs/home/debian/secrets.yml forgejo-v$version/rootfs/home/debian/secrets.yml )

Move the container to replicated storage

lxc-helpers.sh lxc_container_stop $name
sudo mv /var/lib/lxc/$name /precious/lxc/$name
sudo ln -s /precious/lxc/$name /var/lib/lxc/$name
lxc-helpers.sh lxc_container_start $name