A Workaround for Syncing and Updating Multiple Repositories with Flux

Flux is the GitOps Kubernetes operator, which is most useful when used as a deployment tool at the end of a Continuous Delivery pipeline. Flux will make sure that your new container images and config changes are propagated to the cluster.

However, at this moment, flux only works with a single git repository containing Kubernetes manifests.

Let's say you have three applications from three different repositories. If you run fluxctl install for each application on different namespace, and list the controllers with the last namespace you created.

fluxctl list-controllers --k8s-fwd-ns=app3

WORKLOAD                    CONTAINER    IMAGE                                                                                                RELEASE  POLICY
default:deployment/app1     app1         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app1:f8ebcf87b02cd334b4228c1d22fe001dafff9ca6      ready   
default:deployment/app2     app2         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app2:92218e4aeefa8f19f5e9a900bc7d07f38b8622c6      ready    
default:deployment/app3     app3         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app3:a1a8231ff2ac89eb70fc353eeceb2470ee2d0ec3      ready    automated       

If you list the controllers with namespace app1

fluxctl list-controllers --k8s-fwd-ns=app1

There is no workload for it

WORKLOAD                    CONTAINER    IMAGE 

Same as app1

fluxctl list-controllers --k8s-fwd-ns=app2

No workload is expected

WORKLOAD                    CONTAINER    IMAGE 

Therefore, even you make a commit to repo app1 or app2, it never triggers the controller to sync and update the repo. Your deployment would remain unchanged.

To fix it, run

kubectl edit clusterrolebinding.rbac.authorization.k8s.io/flux

You should see

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1beta1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"labels":{"name":"flux"},"name":"flux"},"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"flux"},"subjects":[{"kind":"ServiceAccount","name":"flux","namespace":"app3"}]}
  creationTimestamp: "2020-03-13T16:31:43Z"
  labels:
    name: flux
  name: flux
  resourceVersion: "85027"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/flux
  uid: 202463ba-6548-11ea-a8a2-025c790809a6
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flux
subjects:
- kind: ServiceAccount
  name: flux
  namespace: app3

Since you create app3 at the end, the cluster role binding config is modified when you run fluxctl install.

clusterrolebinding.rbac.authorization.k8s.io/flux configured

If you check out flux RBAC template, you can see there is only one subject.

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: {{ template "flux.clusterRoleName" . }}
  labels:
    app: {{ template "flux.name" . }}
    chart: {{ template "flux.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: {{ template "flux.clusterRoleName" . }}
subjects:
  - name: {{ template "flux.serviceAccountName" . }}
    namespace: {{ .Release.Namespace | quote }}
    kind: ServiceAccount
{{- end -}}
{{- end -}}

Therefore, to allow three applications at the same time, we need to add the missing two.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1beta1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"labels":{"name":"flux"},"name":"flux"},"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"flux"},"subjects":[{"kind":"ServiceAccount","name":"flux","namespace":"app1"}]}
  creationTimestamp: "2020-03-13T16:31:43Z"
  labels:
    name: flux
  name: flux
  resourceVersion: "85027"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/flux
  uid: 202463ba-6548-11ea-a8a2-025c790809a6
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flux
subjects:
- kind: ServiceAccount
  name: flux
  namespace: app1
- kind: ServiceAccount
  name: flux
  namespace: app2
- kind: ServiceAccount
  name: flux
  namespace: app3

Once you save the file, it will update the config in the background. Now we can verify the result.

fluxctl list-controllers --k8s-fwd-ns=app1

WORKLOAD                    CONTAINER    IMAGE                                                                                                RELEASE  POLICY
default:deployment/app1     app1         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app1:f8ebcf87b02cd334b4228c1d22fe001dafff9ca6      ready    automated    
default:deployment/app2     app2         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app2:92218e4aeefa8f19f5e9a900bc7d07f38b8622c6      ready    
default:deployment/app3     app3         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app3:a1a8231ff2ac89eb70fc353eeceb2470ee2d0ec3      ready          

fluxctl list-controllers --k8s-fwd-ns=app2

WORKLOAD                    CONTAINER    IMAGE                                                                                                RELEASE  POLICY
default:deployment/app1     app1         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app1:f8ebcf87b02cd334b4228c1d22fe001dafff9ca6      ready    
default:deployment/app2     app2         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app2:92218e4aeefa8f19f5e9a900bc7d07f38b8622c6      ready    automated    
default:deployment/app3     app3         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app3:a1a8231ff2ac89eb70fc353eeceb2470ee2d0ec3      ready          

fluxctl list-controllers --k8s-fwd-ns=app3

WORKLOAD                    CONTAINER    IMAGE                                                                                                RELEASE  POLICY
default:deployment/app1     app1         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app1:f8ebcf87b02cd334b4228c1d22fe001dafff9ca6      ready    
default:deployment/app2     app2         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app2:92218e4aeefa8f19f5e9a900bc7d07f38b8622c6      ready    
default:deployment/app3     app3         123456789123.dkr.ecr.ap-southeast-1.amazonaws.com/app3:a1a8231ff2ac89eb70fc353eeceb2470ee2d0ec3      ready    automated          

Then when you make a commit to your repo app1, app2 and app3, it should auto release your application and your deployment.yml should be updated by flux with a latest docker image URI.