Logo

Docs

Cluster Discovery

cluster-discovery

cluster-discovery aims to provide a decentralized, peer to peer, kubernetes integrated system to discover of peer belonging to a given network. cluster-discovery relies on IPFS for the peer to peer implementation of the solution.

cluster-discovery is implemented as a kubernetes controller, introducing 3 CRDs:

  • SwarmPool
  • Peer
  • Identity

A Swarmpool represents a technical resource, managing all the technical details of deploying and configuring an IPFS Node. An Identity gives informations about yourself on the network. A Peer is a remote peer Identity. The overall system is decentralized; obviously you need how to identify the network you want to get connected to, and a member of the network, but once done, the boostrap member can leave the network, thanks to the Distributed Hash Table used to persist informations behind the scene.

Quick install

Configure the chart

Set the rbac values in values.yaml according to your needs. A service account is created on Identity creation, the specified rbac will be associated to this service account. The cluster-discovery controller publishes the kubeconfig issued from this service account on the IPFS network.
Therefore, any peer belonging to the network will have those authorizations on the Identity namespace.
Chose the RBAC section content wisely !

Run mode

Once done, install the helm chart:

skaffold run --default-repo=eu.gcr.io/irt-sb

Dev mode

For development purpose, launch the service needed in dev mode with skaffold dev command. dev mode is designed for live reload.

Launch in dev mode with local repository (local kind only):

skaffold dev --default-repo=localhost:5000

Launch in dev mode with gcp repository:

skaffold dev --default-repo=eu.gcr.io/irt-sb

Build Helm package for Kubernetes integration and publish

make publish-helm

cluster-discovery-0.0.x.tgz Helm package should be available in the project root directory and published to cluster-discovery GCP bucket storage.

How to

join the default network

Helm package comes with a default bootstrap node, and default swarm.key (identifying a network). To join this network, proceed as following:

apiVersion: v1
kind: Namespace
metadata:
  name: peer-configuration
---
apiVersion: peerdiscovery.irt-saintexupery.com/v1alpha1
kind: SwarmPool
metadata:
 name: cluster1
 namespace: peer-configuration
spec:
 swarms:
   - name: cluster1
     network:
       loadBalancer:
         loadBalancerIP: auto
         exposedAPI: false
       swarmPort: 4001
       apiPort: 5001
     storage:
       capacity: 1Gi
     IPFSVersion: v0.8.0

In this configuration:

loadBalancerIP With the value set at auto, the cluster-discovery controller will create a Kubernetes L4 load balancer, and let the cloud manager get an IP. You can specify an IP here for the Load Balancer instance if you’ve already provisioned a static IP for the service on your side. exposedAPI The IPFS node API will not be exposed through Load Balancer if value set too false. swarmPort This port value will be used by the load balancer, the headless service and the pod to expose the IPFS swarm port. apiPort This port value will be used eventually by the load balancer (if exposed), the headless service and the pod to expose the IPFS API port. storage Specify the size of the underlying Persistent Volume Storage requested by the controller to store the IPFS DHT. IPFSVersion Version of IPFS that will be used, which is directly related with the tag of docker.io/ipfs/go-ipfs. Only v0.8.0 is supported right now.

create a private network

With loadbalancer :

apiVersion: v1
kind: Namespace
metadata:
  name: private-network-peer-configuration
---
apiVersion: peerdiscovery.irt-saintexupery.com/v1alpha1
kind: SwarmPool
metadata:
 name: cluster1-private
 namespace: private-network-peer-configuration
spec:
 swarms:
   - name: cluster1-private
     swarmDefinition:
      swarmKeySecretRef: private-swarm 
     network:
       loadBalancer:
         loadBalancerIP: auto
         exposedAPI: false
       swarmPort: 4001
       apiPort: 5001
     storage:
       capacity: 1Gi
     IPFSVersion: v0.8.0

With traefik:

apiVersion: v1
kind: Namespace
metadata:
  name: private-network-peer-configuration
---
apiVersion: peerdiscovery.irt-saintexupery.com/v1alpha1
kind: SwarmPool
metadata:
 name: cluster1-private
 namespace: private-network-peer-configuration
spec:
 swarms:
   - name: cluster1-private
     swarmDefinition:
      swarmKeySecretRef: private-swarm 
     network:
       traefik:
         traefikEntryPoint: ipfs
         traefikLoadBalancerServiceName: traefik
         traefikLoadBalancerNamespace: traefik-v2
       swarmPort: 4001
       apiPort: 5001
     storage:
       capacity: 1Gi
     IPFSVersion: v0.8.0

Here, specifying a swarmKeySecretRef: private-swarm will let the controller create a new swarm key. Obviously, at the end of the operation, you’ll be alone in the swarm, as you juste created it.

To extract the generated swarm key, proceed as following:

kubectl get secrets -n private-network-peer-configuration private-swarm -o json | jq '.data["swarm-key"]'

"L2tleS9zd2FybS9wc2svMS4wLjAvCi9iYXNlMTYvCjA4NDljN2NlNGNiODllZjIyODQyZGE1MjI5NWQ3ZjIxMDcyYmYxMjQzZmI2YjdlNTUwNjUzMzM5MmFkYWVhMGM="

This key will be useful to let another peer join your network.

Extract your ID

 kubectl get swarmpools.peerdiscovery.irt-saintexupery.com cluster1-private -n private-network-peer-configuration -o json | jq '.status'

{
  "hash": "PXqJ2FxEz64Aofb/Zx374w==",
  "lastUpdate": "2020-10-09T15:06:45Z",
  "status": {
    "cluster1-private": {
      "Addresses": [
        "/ip4/104.155.27.210/tcp/4001/p2p/QmcnhBjk38swiaohSjBrLrFk7geRGnfFx1dxCY6B7DYAiq"
      ],
      "Name": "cluster1-private",
      "Number of bootstraps": 0,
      "Number of connected peers": 0,
      "Peer Id": "QmcnhBjk38swiaohSjBrLrFk7geRGnfFx1dxCY6B7DYAiq"
    }
  }
}

Among other informations (we can see here that there is no boostrap and no connected peer … which is obvious as we’re the swarm creator here), you can get the Adresses field that can be used as bootstrap for an other cluster

Join a private network

On a other cluster, you can try to join the previously created cluster:

apiVersion: v1
kind: Namespace
metadata:
  name: private-network-peer-configuration
---
apiVersion: v1
kind: Secret
metadata:
  name: private-swarm-key
  namespace:  private-network-peer-configuration
type: Opaque
data:
  swarm-key: L2tleS9zd2FybS9wc2svMS4wLjAvCi9iYXNlMTYvCjA4NDljN2NlNGNiODllZjIyODQyZGE1MjI5NWQ3ZjIxMDcyYmYxMjQzZmI2YjdlNTUwNjUzMzM5MmFkYWVhMGM=
---
apiVersion: peerdiscovery.irt-saintexupery.com/v1alpha1
kind: SwarmPool
metadata:
 name: cluster2-private
 namespace: private-network-peer-configuration
spec:
 swarms:
   - name: cluster2-private
     swarmDefinition:
      swarmKeySecretRef: private-swarm-key
      bootstrapNodeAddresses: ["/ip4/104.155.27.210/tcp/4001/p2p/QmZ9JTRZdiARHzW1x28M51rareFNHtJ8xNohyT8Ewr7sQx"]
     network:
       loadBalancer:
         loadBalancerIP: auto
         exposedAPI: false
       swarmPort: 4001
       apiPort: 5001
     storage:
       capacity: 1Gi
     IPFSVersion: v0.8.0

And checking status on cluster 2:

kubectl get swarmpools.peerdiscovery.irt-saintexupery.com cluster2-private -n private-network-peer-configuration -o json | jq '.status'

{
  "hash": "nuiP9lBR/hBS9FxxkTyzeQ==",
  "lastUpdate": "2020-10-09T15:38:26Z",
  "status": {
    "cluster2-private": {
      "Addresses": [
        "/ip4/34.78.69.228/tcp/4001/p2p/QmRwrdNfviriN1FVRCgYB33GExvE81kvt75bzbG7M2rLGz"
      ],
      "Name": "cluster2-private",
      "Number of bootstraps": 1,
      "Number of connected peers": 1,
      "Peer Id": "QmRwrdNfviriN1FVRCgYB33GExvE81kvt75bzbG7M2rLGz"
    }
  }
}

And checking status on cluster 1:

 kubectl get swarmpools.peerdiscovery.irt-saintexupery.com -n private-network-peer-configuration cluster1-private -o json | jq '.status'

{
  "hash": "PXqJ2FxEz64Aofb/Zx374w==",
  "lastUpdate": "2020-10-09T15:39:38Z",
  "status": {
    "cluster1-private": {
      "Addresses": [
        "/ip4/104.155.27.210/tcp/4001/p2p/QmZ9JTRZdiARHzW1x28M51rareFNHtJ8xNohyT8Ewr7sQx"
      ],
      "Name": "cluster1-private",
      "Number of bootstraps": 0,
      "Number of connected peers": 1,
      "Peer Id": "QmZ9JTRZdiARHzW1x28M51rareFNHtJ8xNohyT8Ewr7sQx"
    }
  }
}

We can see here that each peer know an other peer (so they know each other), but only one of them as a bootstrap peer. Bootstrap peer list get actualized when an Identity is published on the IPFS network.

Publish an identity on a network

On cluster 1, create following Kubernetes object:

apiVersion: peerdiscovery.irt-saintexupery.com/v1alpha1
kind: Identity
metadata:
  name: cluster1-identity
  namespace: private-network-peer-configuration
spec:
  swarms:
  - namespace: private-network-peer-configuration
    swarmPool: cluster1-private
    swarm: cluster1-private
  description: I'm cluster 1
  name: Cluster 1
  URL: https://35.240.24.118
  categories: 
    - name: "computing"
    - name: "EO data"
    - name: "IOT data"

On cluster 2, create following Kubernetes object:

apiVersion: peerdiscovery.irt-saintexupery.com/v1alpha1
kind: Identity
metadata:
  name: cluster2-identity
  namespace: private-network-peer-configuration
spec:
  swarms:
  - namespace: private-network-peer-configuration
    swarmPool: cluster2-private
    swarm: cluster2-private
  description: I'm cluster 2
  name: Cluster 2
  URL: https://35.187.98.45
  categories: 
    - name: "computing"
    - name: "sentinel S3A"
    - name: "weather forecast"

List the peers

After a while, on both cluster, you should see the identity of the other cluster appearing as an instance of Peer

On cluster 1:

kubectl get peers -A

NAMESPACE                            NAME                                             PEERNAME    STATUS   IP                     DESCRIPTION
private-network-peer-configuration   qmpgihqyzm2hq9k7n2mrrfhjtauzmdhqnp8sbrgaw2mkd1   Cluster 2   Online   https://35.187.98.45   I'm cluster 2

To get the full details:

plm@B20191872:~|⇒  kubectl describe peer qmpgihqyzm2hq9k7n2mrrfhjtauzmdhqnp8sbrgaw2mkd1 -n private-network-peer-configuration

Name:         qmpgihqyzm2hq9k7n2mrrfhjtauzmdhqnp8sbrgaw2mkd1
Namespace:    private-network-peer-configuration
[...]
Spec:
  Categories:
    Name:       computing
    Name:       sentinel S3A
    Name:       weather forecast
  Description:  I'm cluster 2
  Name:         Cluster 2
  URL:          https://35.187.98.45
Status:
  Missing Counter:  0
  Status:           Online
Events:             <none>

On cluster 2:

kubectl get peers -A

NAMESPACE                            NAME                                             PEERNAME    STATUS   IP                      DESCRIPTION
private-network-peer-configuration   qmz9jtrzdiarhzw1x28m51rarefnhtj8xnohyt8ewr7sqx   Cluster 1   Online   https://35.240.24.118   I'm cluster 1

To get the full details:

kubectl describe peer qmz9jtrzdiarhzw1x28m51rarefnhtj8xnohyt8ewr7sqx -n private-network-peer-configuration

Name:         qmz9jtrzdiarhzw1x28m51rarefnhtj8xnohyt8ewr7sqx
Namespace:    private-network-peer-configuration
[...]
Spec:
  Categories:
    Name:       computing
    Name:       EO data
    Name:       IOT data
  Description:  I'm cluster 1
  Name:         Cluster 1
  URL:          https://35.240.24.118
Status:
  Missing Counter:  0
  Status:           Online
Events:             <none>

Update an Identity

Change cluster 1 Identity as following:

apiVersion: peerdiscovery.irt-saintexupery.com/v1alpha1
kind: Identity
metadata:
  name: cluster1-identity
  namespace: private-network-peer-configuration
spec:
  swarms:
  - namespace: private-network-peer-configuration
    swarmPool: cluster1-private
    swarm: cluster1-private
  description: I'm cluster 1, but updated
  name: Cluster 1
  URL: https://35.240.24.118
  categories: 
    - name: "computing"
    - name: "EO data"
    - name: "IOT data"
    - name: "new category"

You should see the update in the Peer description in the cluster 2:

 kubectl describe peer qmz9jtrzdiarhzw1x28m51rarefnhtj8xnohyt8ewr7sqx -n private-network-peer-configuration

Name:         qmz9jtrzdiarhzw1x28m51rarefnhtj8xnohyt8ewr7sqx
Namespace:    private-network-peer-configuration
[...]
Spec:
  Categories:
    Name:       computing
    Name:       EO data
    Name:       IOT data
    Name:       new category
  Description:  I'm cluster 1, but updated
  Name:         Cluster 1
  URL:          https://35.240.24.118
Status:
  Missing Counter:  0
  Status:           Online
Events:             <none>

Delete an identity

On cluster 2:

kubectl delete -f cluster2_private_network_identity.yaml

identity.peerdiscovery.irt-saintexupery.com "cluster2-identity" deleted

After a while:

kubectl get peers -A

No resources found

Get the kubeconfig associated to a Peer

From cluster 1:


# We pickubectl up the first available peer here

PEER=$(kubectl get peers -n peer-configuration -o json | jq -r '.items[0].metadata.name')
kubectl get secret $(kubectl get peer ${PEER} -n peer-configuration -o json | jq -r '.metadata.annotations["clusterdiscovery/kubeconfig-secret"]') -n peer-configuration -o json | jq -r '.data["config"]' | base64 -d > /tmp/peerconfig.yaml
kubectl --kubeconfig /tmp/peerconfig.yaml get pods

NAME                            READY   STATUS    RESTARTS   AGE
public-c2-sp-public-c2-s-ss-0   1/1     Running   0          60m

From cluster 2:


# We pickubectl up the first available peer here

PEER=$(kubectl get peers -n public-networkubectl -o json | jq -r '.items[0].metadata.name')
kubectl get secret $(kubectl get peer ${PEER} -n public-networkubectl -o json | jq -r '.metadata.annotations["clusterdiscovery/kubeconfig-secret"]') -n public-networkubectl -o json | jq -r '.data["config"]' | base64 -d > /tmp/c1config.yaml
kubectl --kubeconfig /tmp/c1config.yaml get pods

NAME                             READY   STATUS    RESTARTS   AGE
cluster1-cluster1-ss-0   1/1     Running   0          65m

If RBAC configured in values.yaml doesn’t allow to get secrets on cluster 1 side:

kubectl --kubeconfig /tmp/c1config.yaml get secrets

Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount:peer-configuration:cluster1-identity" cannot list resource "secrets" in API group "" in the namespace "peer-configuration"

Get business metrics from controller

The cluster-discovery controller publishes metrics about the total number of peers and the total number of synchronized peers (with valid published identity) per swarm. Those informations are accessible via the /metrics endpoint protected by the kube-rbac-proxy side container.

Metrics are formatted for prometheus/grafana integration.

Example:

Create the proper role (binded to default service account in default namespace for the following example, set it up to the prometheus SA for proper integration):

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 name: metrics-reader
rules:
- nonResourceURLs:
 - "/metrics"
 verbs:
 - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
   name: metrics-reader-binding
   namespace: default
roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
   name: metrics-reader
subjects:
   - kind: ServiceAccount
     name: default
     namespace: default

Spawn a curl pod in default namespace:

kubectl run curl --image=radial/busyboxplus:curl -i --tty --rm

curl -k -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://cluster-discovery-controller-manager-metrics-service.cluster-discovery:8443/metrics | grep cluster

# HELP cluster_discovery_controller_identity_controller_peers_number Number of known peer associated to a given identity
# TYPE cluster_discovery_controller_identity_controller_peers_number gauge
cluster_discovery_controller_identity_controller_peers_number{identity_name="cluster1-identity",identity_namespace="private-network-peer-configuration",status="Out of sync",swarm_name="cluster1-private",swarmpool_name="cluster1-private",swarmpool_namespace="private-network-peer-configuration"} 0
cluster_discovery_controller_identity_controller_peers_number{identity_name="cluster1-identity",identity_namespace="private-network-peer-configuration",status="Synchronized",swarm_name="cluster1-private",swarmpool_name="cluster1-private",swarmpool_namespace="private-network-peer-configuration"} 1
# HELP cluster_discovery_controller_identity_controller_sync_peers_number Number of synchronized peers (peer with a valid published identity) associated to a given identity
# TYPE cluster_discovery_controller_identity_controller_sync_peers_number gauge
cluster_discovery_controller_identity_controller_sync_peers_number{identity_name="cluster1-identity",identity_namespace="private-network-peer-configuration",status="Out of sync",swarm_name="cluster1-private",swarmpool_name="cluster1-private",swarmpool_namespace="private-network-peer-configuration"} 0
cluster_discovery_controller_identity_controller_sync_peers_number{identity_name="cluster1-identity",identity_namespace="private-network-peer-configuration",status="Synchronized",swarm_name="cluster1-private",swarmpool_name="cluster1-private",swarmpool_namespace="private-network-peer-configuration"} 0