Swap to v4 impl

README.md

@@ -3,119 +3,21 @@ My cluster configuration that serves to automate the deployment and handling of
 
 I'll try to include any pertinent documentation here in the tooling I use or the setup.
 
+#### This uses a very much alpha build of my pulumi-cljs library
 
 #### Upcoming
-Break this into three repos. IaC, the Pulumi CLJS library, and my dot files. We'll also be moving data from our old instances to the new IaC managed cluster. c:
-Current roadmap for that is breaking apart the Vault provider into its actual core components as it is currently an anti-pattern in the way it combines multiple provider functionalities through it. It relying on the config file even is a bit of an issue.
-Furthermore, we are unable to effectively destructure secrets in the execute function in the current design. However, since we'd want to change to remove the anti-pattern mentioned above, we'd ideally actually just reference secrets through the vault resource output from the given resource config's stack execution.
+Currently only want to expand on making the final service declarations functional.
+I'll be revising the library to better incorporate changes I'd like to see and I'd like to make the core cleaned up further.
+In the more immediate that'd be:
+- Simplifying the declarations
+- Pulling out the library functions
+- Cleaning up how re-used configs are appended to
 
-get-provider-outputs-config inside utils.providers.cljs currently runs under the expectation that shared stack already exists... which inherently is flawed on an initial run. Will need to revise a little. Similarly get-stack-refs works on the same flawed principle.
-Maybe we can move them to stack definitions (which currently exist in base.cljs). I think in an ideal design we could actually inherently scope the entire thing out. I'm inspired by how Guix allows system definitions to be written, and there isn't anything stopping a large block for a stack being like:
-```
-(def some-stack
-{:stack-registry
- [{:stack [:k8s:secret :k8s:chart]
-   :app-namespace "kube-system"
-   :app-name "hcloud-csi"
-   :vault-load-yaml false
-   :k8s:secret-opts {:metadata {:name "hcloud"
-                            :namespace "kube-system"}
-                 :stringData {:token  (-> cfg :hcloudToken)}}
-   :k8s:chart-opts {:fetchOpts {:repo "https://charts.hetzner.cloud"}
-                :values {:controller {:enabled false
-                                      :existingSecret {:name "hcloud-csi-secret"}
-                                      :node {:existingSecret {:name "hcloud-csi-secret"}}}}}}
-    {:stack [:vault:prepare :docker:image :k8s:secret :k8s:chart]
-   :app-namespace "caddy-system"
-   :app-name      "caddy-ingress-controller"
-   :k8s:image-port 8080
-   :k8s:vault-load-yaml false
-   :k8s:image-opts {:imageName '(str repo "/" app-name ":latest")}
-   :docker:image-opts {:registry {:server (-> cfg :public-image-registry-url)
-                                  :username (-> cfg :public-image-registry-username)
-                                  :password (-> cfg :public-image-registry-password)}
-                       :tags [(str (-> cfg :public-image-registry-url) "/" (-> cfg :public-image-registry-username) "/" "caddy")]
-                       :push true}
-   :k8s:chart-opts {:fetchOpts {:repo "https://caddyserver.github.io/ingress"}
-                    :values
-                    {:ingressController
-                     {:deployment {:kind "DaemonSet"}
-                      :daemonSet {:useHostPort true}
-                      :ports {:web {:hostPort 80}
-                              :websecure {:hostPort 443}}
-                      :service {:type "NodePort"
-                                :externalTrafficPolicy "Local"}
-                      :image {:repository 'repo
-                              :tag "latest"}
-                      :config {:email 'email}}}}}
-    ]
-    :stack-references { :init (new pulumi/StackReference "init") 
-                        :shared (new pulumi/StackReference "shared")}
-    :provider-configs {:harbor {:stack :shared
-                :outputs ["username" "password" "url"]}}
-    })
-```
-and that effectively defines an entire stack and is executable on (with the option to scope out to files to reduce the sheer verbosity in a single)
-In that regard, I think we've made decent headway in achieving a similar design and behavior where a config should provide reproducible results.
-Due to the nature of npm packages, it is a bit hard to *lock* to certain package versions as easily.
+Eventually the user will have more programmatic choice of execution too so as to have a program that ingests the library first and applies their stack defaults to it.
+The only limitation there would be that it does limit multi-cloud designs perhaps. Though for those, the unconsumed library would still be very much a practical option.
 
-DNS should be swapped with a Cloudflare provider instead and more appropriately allow EACH service to plainly define a DNS entry.
+To be more clear eventually rewriting the specs for defaults to match whatever stack configs given is likely the most optimal choice. Basically utilizing the homoiconic nature of Clojure and consuming the first program to generate the final one. Stubbing in the replacements as we walk through it.
 
-Local config loading or something should also be a provider, as obviously we would want to be able to pass through virtually anything to a service. That way they can be accessed later (this would replace the weird load-yaml that is a leftover from prior iterations)
-
-pulumi2crd should perhaps include some install instructions and some insight into usage
-Currently the script builds correctly, but since they are version dependant we might want to have some sort of version management for each of these components. That way they can be updated in a similar mechanism to a normal npm package. We can make CICD pipelines for them for this with some sort of cron scheduling. Emulating Renovate behavior (or we can see if Renovate can be useful here even).
-
-Those generated CRDs should not be baked into the provider utils but instead be treated as an expansion. This way it is neatly organized into official and extended functionality.
-
-Default values (like in K8) are opinionated. They do need to outline how to use a structure for example, but it should also be convenient to use any other resource like Nginx instead of Traefik or Azure instead of Cloudflare. A macro could be applied to them (preferably after their declaration, so their default state remains opinionated) to swap out which provider an individual chooses to use. It can be an added field in the core declarations for processing. Obvious goal for this is expansiveness. There should be clean, reusable defaults and everything should be easily modifiable and expandable.
-
-
-Resource declarations might benefit from being *able* to splinter when needed. Currently they are VERY MUCH locked to a singleton pattern. While we can "loop" over stuff inside a declaration it still only ever makes *one* resource.
-
-Currently, certificates relies upon a prior step existing and that in itself is a bit of an anti-pattern... So in the future our options NEED some way of informing the resolver and deployer that it has custom execution.
-```
-:k8s:certificates
-   {:constructor (.. cert-manager -v1 -Certificate)
-    :provider-key :k8s
-    :defaults-fn (fn [env]
-                   (p-> env :options :vault:prepare "stringData" .-domains
-                        #(vec
-                          (for [domain (js/JSON.parse %)] 
-                            (let [clean-name (clojure.string/replace domain #"\." "-")]
-                              {:_suffix clean-name
-                               :spec {:dnsNames [domain (str "*." domain)]
-                                      :secretName (str clean-name "-tls")}})))))}
-```
-The above is unideal. I think the best path forward for that is an override?  Considering that some might not use Vault.
-It might, instead, benefit from a high level user declaration of intent regarding the location of their secrets/settings. I mentioned above to have it resolve based on what providers utilized (within reason for support). That removes the inherent reliance, but it still does leave resolution in the default-fn in an unideal manner. It doesn't work to make top-level functions resolve on the outer layer as the Vault entry wouldn't exist yet. 
-If we do the user intent, we can at least change it to be a standard such as 
-```
-(p-> env :options :secrets .-domains #(function here))
-```
-I should add that this function would be in a more *plugin* since it isn't inherently a built-in for K8s. Same for Gateway.
-
-It wouldn't hurt to add some extension for developing these too. Increasing clarity on manner of declaration can not hurt.
-
-Should also revise default-fn to recursively call certificate and just allow the default-fn to unwind the values.
-
----
-It may be helpful to redesign the stack mechanism entirely so that resources and such are declared like:
-(def config
-    {:stack [
-        {:item-name 
-            {:options-in-here}} 
-        {:item-name-2 
-            {:options-in-here}}
-    ]})
-Where this provides much clearer association and each resource has its options readily available. As such you could declare duplicate keys in the same config. It would make resource associations much more explicit and cleaner written.
-It would require a decent amount of revision, so no rush on this.
-
-
-Mentioned above a bit, but eventually rewriting the specs for defaults to match whatever stack configs given is likely the most optimal choice. Basically utilizing the homoiconic nature of Clojure and consuming the first program to generate the final one. Stubbing in the replacements as we walk through it.
-
-
-Component spec really needs to be moved out of stack_processor as it is just such a large block of data that so better belongs w/ the providers themselves.
 
 ---
 
@@ -125,7 +27,6 @@ The long term goal is for this to be a mostly uninteractive, to completion set u
 More immediately, as we've closed in on a functional end-to-end alpha build and learned several choices we could've made to better design a next build, we'll actually use this to move our services off a single VPS w/ a docker compose and into a cluster fully generated by this with no setup or involvement on our part.
 
 
-
 ### Initial requirements
 #### Need to Revise as we swapped to using Pulumi Automation API so the entire process is automated
 

deps.edn

@@ -1,8 +1,9 @@
 
 {:paths ["src/main"]
-;;:deps {gigiaj/pulumicljs {:local/root "../pulumi-clojurescript"}}
- :deps {gigiaj/pulumicljs
-        {:git/url "https://github.com/GigiaJ/pulumi-clojurescript.git"
+ :deps {
+        gigiaj/pulumicljs {:local/root "../pulumi-clojurescript"}
+        #_gigiaj/pulumicljs
+        #_{:git/url "https://github.com/GigiaJ/pulumi-clojurescript.git"
          :git/sha "01133c6b412db9d6541bc8566dd0e6fe3ffd514c"
          }
         funcool/promesa {:mvn/version "11.0.678"}

src/main/infra/openbao.cljs

@@ -274,7 +274,7 @@
    :app-name "openbao"
    :exec-fn  execute-fn
    :vault-load-yaml false
-   :k8s:chart-opts {:fetchOpts {:repo "https://openbao.github.io/openbao-helm"}
+   :k8s:chart-opts {:repositoryOpts {:repo "https://openbao.github.io/openbao-helm"}
                 :transformations [(fn [props opts]
                                    (let [kind (:kind props)]
                                      (if (= kind "StatefulSet")

src/main/k8s/add_ons/cert_manager.cljs

@@ -5,7 +5,7 @@
    :app-namespace "cert-manager"
    :app-name "cert-manager"
    :is-prod? true
-   :k8s:chart-opts {:fetchOpts {:repo "https://charts.jetstack.io"}
+   :k8s:chart-opts {:repositoryOpts {:repo "https://charts.jetstack.io"}
                     :chart "cert-manager"
                     :version "v1.15.0"
                     :namespace "cert-manager"

src/main/k8s/add_ons/crd/cert_manager.cljs

@@ -0,0 +1,7 @@
+(ns k8s.add-ons.crd.cert-manager)
+
+(def config
+  {:stack [:k8s:config-file]
+   :app-name "cert-manager"
+   :version "v1.15.0"
+   :k8s:config-file-opts {:file '(str "https://github.com/cert-manager/cert-manager/releases/download/" version "/cert-manager.crds.yaml")}})

src/main/k8s/add_ons/crd/gateway_api.cljs

@@ -0,0 +1,7 @@
+(ns k8s.add-ons.crd.gateway-api)
+
+(def config
+  {:stack [:k8s:config-file]
+   :app-name "gateway-api"
+   :version "v1.4.0"
+   :k8s:config-file-opts {:file '(str "https://github.com/kubernetes-sigs/gateway-api/releases/download/" version "/experimental-install.yaml")}})

src/main/k8s/add_ons/crd/traefik.cljs

@@ -0,0 +1,10 @@
+(ns k8s.add-ons.crd.traefik)
+
+(def config
+  {:stack [:k8s:chart]
+   :app-name "traefik-crds"
+   :k8s:chart-opts {:repositoryOpts {:repo "https://traefik.github.io/charts"}
+                    :chart "traefik-crds"
+                    :version "v1.12.0"
+                    :namespace "traefik" 
+                    :values {:deleteOnUninstall true}}})

src/main/k8s/add_ons/csi_driver/hetzner.cljs

@@ -10,7 +10,7 @@
    :k8s:secret-opts {:metadata {:name "hcloud"
                             :namespace "kube-system"}
                  :stringData {:token  (-> cfg :hcloudToken)}}
-   :k8s:chart-opts {:fetchOpts {:repo "https://charts.hetzner.cloud"}
+   :k8s:chart-opts {:repositoryOpts {:repo "https://charts.hetzner.cloud"}
                 :values {:controller {:enabled false
                                       :existingSecret {:name "hcloud-csi-secret"}
                                       :node {:existingSecret {:name "hcloud-csi-secret"}}}}}})

src/main/k8s/add_ons/csi_driver/wasabi.cljs

@@ -7,7 +7,7 @@
    :no-namespace true
    :app-name      "wasabi-csi"
    :k8s:chart-opts {:chart "csi-s3"
-                :fetchOpts {:repo "https://yandex-cloud.github.io/k8s-csi-s3/charts"}
+                :repositoryOpts {:repo "https://yandex-cloud.github.io/k8s-csi-s3/charts"}
                 :values {:controller {:enabled false
                                                         :existingSecret {:name "wasabi-csi-secrets"}
                                                         :node {:existingSecret {:name "wasabi-csi-secrets"}}}}

src/main/k8s/add_ons/gateway/traefik.cljs

@@ -1,20 +1,42 @@
 (ns k8s.add-ons.gateway.traefik)
 
 (def config
-  {:stack [:vault:prepare [:k8s :secret :chart :gateway :certificates]]
+  {:stack [:vault:prepare [:k8s :secret :chart :gateway-class :gateway :certificates]]
    :app-namespace "traefik"
    :app-name "traefik"
    :is-prod? true
    :vault-load-yaml false
-   :k8s:chart-opts {:fetchOpts {:repo 'repo}
+   :k8s:chart-opts {:skipCrds true
+                    :repositoryOpts {:repo 'repo}
                     :chart 'chart
+                    :transformations  [(fn [args _opts] (let [kind (get-in args [:resource :kind])]
+                                       (if (= kind "CustomResourceDefinition")
+                                         nil
+                                         args)))]
                     :version "37.3.0"
                     :namespace "traefik"
                     :values {:providers {:kubernetesGateway {:enabled true}}
-                             :gatewayClass {:enabled true
-                                            :name "traefik"}}}
+                             :gatewayClass {:enabled false}
+                             :gateway {:enabled false}
+                             :ports {:web {:port 8000
+                                           :expose {:default true}
+                                           :exposedPort 80
+                                           :protocol "TCP"}
+
+                                     :websecure {:port 8443
+                                                 :expose {:default true}
+                                                 :exposedPort 443
+                                                 :protocol "TCP"
+                                                 :transport {:respondingTimeouts
+                                                             {:readTimeout "600s"
+                                                              :writeTimeout "600s"
+                                                              :idleTimeout "600s"}}}}}}
    :k8s:gateway-opts
    {:metadata {:name "main-gateway"
                :namespace "traefik"}
     :spec {:gatewayClassName "traefik"
-           :listeners '(make-listeners domains)}}})
+           :listeners '(make-listeners domains)}}
+   
+   :k8s:gateway-class-opts
+   {:spec {:controllerName "traefik.io/gateway-controller"}}
+   })

src/main/k8s/add_ons/image_registry/harbor.cljs

@@ -6,7 +6,7 @@
    :app-name "harbor"
    :image-port 80
    :vault-load-yaml false
-   :k8s:chart-opts {:fetchOpts {:repo "https://helm.goharbor.io"}
+   :k8s:chart-opts {:repositoryOpts {:repo "https://helm.goharbor.io"}
                     :values {:externalURL '(str "https://" host)
                              :expose {:type "route"
                                       :tls {:enabled false}

src/main/k8s/add_ons/ingress_controller/caddy.cljs

@@ -13,7 +13,7 @@
                                   :password (-> cfg :public-image-registry-password)}
                        :tags [(str (-> cfg :public-image-registry-url) "/" (-> cfg :public-image-registry-username) "/" "caddy")]
                        :push true}
-   :k8s:chart-opts {:fetchOpts {:repo "https://caddyserver.github.io/ingress"}
+   :k8s:chart-opts {:repositoryOpts {:repo "https://caddyserver.github.io/ingress"}
                     :values
                     {:ingressController
                      {:deployment {:kind "DaemonSet"}

src/main/k8s/add_ons/secret_replicator.cljs

@@ -6,4 +6,4 @@
    :no-namespace true
    :app-namespace "kube-system"
    :app-name      "kubernetes-replicator"
-   :k8s:chart-opts {:fetchOpts {:repo "https://helm.mittwald.de"}}})
+   :k8s:chart-opts {:repositoryOpts {:repo "https://helm.mittwald.de"}}})

src/main/k8s/services/nextcloud/service.cljs

@@ -6,7 +6,7 @@
    :app-name      "nextcloud"
    :image-port 8080
    :vault-load-yaml true
-   :chart-opts {:fetchOpts {:repo "https://nextcloud.github.io/helm/"}
+   :chart-opts {:repositoryOpts {:repo "https://nextcloud.github.io/helm/"}
                 :values {:nextcloud {:host 'host
                                      :trustedDomains ['host 'app-name]}}
                 :transformations (fn [args _opts]

src/main/k8s/services/prometheus/service.cljs

@@ -7,7 +7,7 @@
    :image-port 8080
    :vault-load-yaml true
    :chart-opts {:chart "kube-prometheus-stack"
-                :fetchOpts {:repo "https://prometheus-community.github.io/helm-charts"}
+                :repositoryOpts {:repo "https://prometheus-community.github.io/helm-charts"}
                 :namespace "monitoring"
                 :values {:grafana {:adminPassword 'password
                                    :ingress {:enabled true

src/main/k8s/services/renovate/service.cljs

@@ -8,7 +8,7 @@
    :image-port    8080
    :vault-load-yaml true
    :chart-opts
-   {:fetchOpts {:repo "https://docs.renovatebot.com/helm-charts"}
+   {:repositoryOpts {:repo "https://docs.renovatebot.com/helm-charts"}
     :values
     {:renovate
      {:config {:platform "github"

src/main/stack_resource_definitions.cljs

@@ -6,14 +6,14 @@
    [infra.dns :as dns]
    [k8s.preparers.harbor :as harbor-prepare]
    
-   [k8s.add-ons.ingress-controller.caddy :as caddy]
    [k8s.add-ons.gateway.traefik :as traefik]
    [k8s.add-ons.cert-manager :as cert-manager]
+   [k8s.add-ons.crd.cert-manager :as cert-manager-crd]
+   [k8s.add-ons.crd.gateway-api :as gateway-api-crd]
+   [k8s.add-ons.crd.traefik :as traefik-crds]
    [k8s.add-ons.csi-driver.wasabi :as wasabi-csi]
    [k8s.add-ons.image-registry.harbor :as harbor]
    [k8s.add-ons.secret-replicator :as secret-replicator]
-   [k8s.add-ons.minio :as minio]
-   [k8s.add-ons.s3proxy :as s3proxy]
    [k8s.add-ons.proxy :as proxy]
    [k8s.services.nextcloud.service :as nextcloud-service]
    [k8s.services.mesite.service :as mesite-service]
@@ -51,8 +51,13 @@
 
 (def shared-resources-definition
   (create-resource-definition
-   [traefik/config cert-manager/config
-    dns/config wasabi-csi/config proxy/config secret-replicator/config 
+   [dns/config
+    cert-manager-crd/config
+    gateway-api-crd/config
+    traefik-crds/config
+    cert-manager/config
+    traefik/config
+    wasabi-csi/config proxy/config secret-replicator/config 
     harbor/config
     ]
    ["base" "init"]
@@ -67,7 +72,7 @@
 
 (def deployment-resources-definition
   (create-resource-definition
-   [nextcloud-service/config foundryvtt-service/config mesite-service/config productive-service/config gitea-service/config act-runner-service/config]
+   [#_nextcloud-service/config foundryvtt-service/config mesite-service/config productive-service/config gitea-service/config act-runner-service/config]
    ["base" "init" "shared"]
    (general-provider-output-refs)))