Setting up a multi node Hyperledger Fabric network with Docker Swarm | Skcript


(S//Bot) #1

Hyperledger Fabric is now getting very popular and the community is coming out with various deployment strategies for the production. One such implementation is the docker swarm method, mainly when you go with multi-node architecture which is likely to be the case for a proper blockchain network. In this article, we’ll be seeing how we can do the docker swarm setup for our Hyperledger Fabric network.

Prerequisite

Before proceeding into the article, I want to make sure that you are much aware of how hyperledger fabric works and what architecture is behind it to understand this workflow much better and with less confusion.

And make sure you use a ubuntu machine for deploying this, since it won’t work in mac as docker swarm works little different with MacOS.

  • 3 Machines/VM with Ubuntu 16.04
  • Docker
  • Docker Compose

Pulling Docker Images for Hyperledger Fabric

In our tutorial we’re using Hyperledger Fabric 1.1 and the requirement. I never tested with 1.2 but it should work as expected. To make it more smooth, make sure you’re using 1.1 as the image. In order to do that pull the 1.1 tag.

docker pull hyperledger/fabric-tools:x86_64-1.1.0 docker pull hyperledger/fabric-orderer:x86_64-1.1.0 docker pull hyperledger/fabric-peer:x86_64-1.1.0 docker pull hyperledger/fabric-javaenv:x86_64-1.1.0 docker pull hyperledger/fabric-ccenv:x86_64-1.1.0 docker pull hyperledger/fabric-ca:x86_64-1.1.0 docker pull hyperledger/fabric-couchdb:x86_64-0.4.6

Finally tag them as the latest images to make sure when docker-compose, runs them corresponding images are used.

docker tag hyperledger/fabric-ca:x86_64-1.1.0 hyperledger/fabric-ca:latest docker tag hyperledger/fabric-tools:x86_64-1.1.0 hyperledger/fabric-tools:latest docker tag hyperledger/fabric-orderer:x86_64-1.1.0 hyperledger/fabric-orderer:latest docker tag hyperledger/fabric-peer:x86_64-1.1.0 hyperledger/fabric-peer:latest docker tag hyperledger/fabric-javaenv:x86_64-1.1.0 hyperledger/fabric-javaenv:latest docker tag hyperledger/fabric-ccenv:x86_64-1.1.0 hyperledger/fabric-ccenv:latest docker tag hyperledger/fabric-couchdb:x86_64-0.4.6 hyperledger/fabric-couchdb:latest

Infrastructure

In this article, we’ll be looking at a Fabric network with 3 organizations installed in 3 physical machines, and since it’s going to be a production deployment steps we’ll be using Kafka based ordering service with 3 orderers (one per each organization). The infrastructure will also include 1 Fabric CA per organization and the couchdb as world state in each and every peer.

We’ll be having one channel called mychannel and a chaincode named mychaincode installed in the channel.

What is Docker Swarm?

Docker swarm is an amazing tool by docker for clustering and scheduling the docker containers in your infrastructure. It helps you to orchestrate the network by having various disaster recovery mechanism for high availability and it looks to be very good for a blockchain network.

In a docker swarm you can create a cluster of nodes which will be able to communicate between then with help of a overlay docker network. The nodes are categorised into worker nodes and leader nodes. The leader nodes are the one which have control over the network and they are responsible for load balancing and container recovery in case of disaster. While the worker nodes are the nodes which will help with running the containers.

In our use case, we’ll be using 3 physical machines out of which 1 will be the leader and the other two will be worker nodes. So we assume that the leader node can be the node of the blockchain owner who has the rights over the private blockchain.

Challenges

When we try to deploy a fabric network in a docker swarm network, we’ll face the following issues or challenges.

  1. When we deploy set of containers, by default it pushes the containers to the nodes in the network in a random fashion. But in hyperledger fabric case we are going to map each and every organization to a specific node which might be owned by the corresponding organization. So we should be careful in handling node targets before deployments.
  2. The certificates for each and every container is usually mapped relative to the docker compose config YML file, but in docker swarm where we deploy and push to different servers we can’t use the relative path and can use only the absolute path. So in order to address this issue, we’ll be choosing a specific path in each VM or machine where the certs will be hosted and we’ll be using that in the docker compose files.

Getting Started

In order to make this deployment much more simpler, we’ve created scripts and configuration which we’ll use in this article’s step.

Creating VMs or Physical Machines.

As the first step, create 3 VMs in services like AWS, GCP, or setup the physical machines with Ubuntu 16.04. Once you’ve the machines ready, ssh into them and install the docker and docker compose.

Clone the repository.

As I mentioned earlier we’ve created a sample project which has the scripts and config for Hyperledger Fabric to run in 3 machines. You can clone that repo in all of your machines.

git clone https://github.com/skcript/hlf-docker-swarm

Once you clone it, you can see two folders, one has the sample chaincode (Fabcar) and the other which has the scripts and configuration for the docker swarm network. In this article, we’ll mostly use the scripts that is already written.

Understanding the configurations

Most of the configurations are similar to the single node deployment scripts. The key things to notice here are the docker compose files and the set of configuration that are modified for deploying the services in the swarm node.

Docker Compose Configurations for Swarm

When you docker swarm, we have lot of constraints in the compose files. Under each service block, we have to define the deploy configuration which has configurations like replication count, restart policy when the node crashes, and also the key important factor which says in which worker/manager node the service has to be deployed. This is important for a hyperledger fabric network since it invokes certs to be given to specific VM or physical machines in the network. We use swarm hostnames to specify it. The hostnames can be identified with help of following command

deploy: replicas: 1 restart_policy: condition: on-failure delay: 5s max_attempts: 3 placement: constraints: - node.hostname == <MACHINE_HOSTNAME>

And when you define the network in each compose-file, you have to provide it as a overlay network.

networks: skcript: external: name: skcript

In our network sample, you’ll be seeing this in all the docker compose files.

Mounting the volumes

In a usual case with single machine deployments, we mount the certs to provide it inside the container and most cases it’ll be a relative system path from the directory of compose files. But in case of swarm network we can use the relative paths since the containers are deployed from one manager node to any of the worker nodes, thus when the deployment in distributed system happens, the relative path will be meaningless.

Thus you’ve to use the absolute path only for any volume and make sure that they are present in the corresponding machines. For an instance in our orderer_org1 service we’ve mounted the following

volumes: - /var/mynetwork/certs/crypto-config/ordererOrganizations/example.com/users:/var/hyperledger/ - /var/mynetwork/certs/crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/msp:/var/hyperledger/msp - /var/mynetwork/certs/crypto-config/ordererOrganizations/example.com/orderers/orderer0.example.com/tls:/var/hyperledger/tls - /var/mynetwork/certs/config/:/var/hyperledger/config

And the host path is an absolute path point to folders inside /var/mynetwork/ .So make sure that these paths are available in the node org1 which is defined in the deployment policy.

Deployment

Now let’s get into the deployment process. Once you cloned the repository, open it in the terminal. Also make sure you’ve 3 machines ready for a multi host deployment.

cd hlf-docker-swarm/network

Here we’ve a set of predefined scripts that you have to run in order to setup the environment for docker swarm.

Step 1 : Setup Swarm Network

As the first step, we’ve to create a new swarm service.

docker swarm leave -f docker swarm init

The above commands will leave if the node joined any other docker swarm and then create a new swarm.

When you run it, you’ll receive a worker join token which you need to run in the other nodes in the network.

As mentioned above, you need to take this token and run it in other nodes that are to be joined in the network.

Once you make other nodes join, you can use the following commands to verify it.

And you see all the nodes joined in the network along with their hostnames.

Alternatively, We also have a single script to run these above steps.

Step 2: Create Overlay Network

Once you’ve created the swarm and joined the nodes, now we’ve to create a overlay network that will be used to communicate in the swarm. In this article, I’ll be naming the network name as Skcript. Run the following command to create a overlay network:

docker network create --driver overlay --subnet=10.200.1.0/24 --attachable skcript

Or, you can use my script which does the same.

Now if you do docker network ls you’ll be able to see the newly created overlay network

Step 3 : Node configuration and moving crypto certs

As we have the docker swarm and the network ready with all the nodes joined. It’s time to configure our docker compose files to make it deploy in the right nodes and also we’ve to move the certs to all the machine’s corresponding path.

Before node configuration, make sure you’ve the certs moved to the absolute path that is mentioned in the volumes. Clone the repo in all the machines and run the following command which will move the certs to the corresponding folder.

In order to configure the node’s hostname, you’ve to edit the .env file that is located in the root of network folder.

Update the following variables to corresponding hostname of machines. You must reference the hostname of nodes from the above mentioned results.

ORG1_HOSTNAME=<ORG1 HOSTNAME> ORG2_HOSTNAME=<ORG1 HOSTNAME> ORG2_HOSTNAME=<ORG1 HOSTNAME>

After editing, save it and close. Now you’ve to run our script to update this in all our compose files. In order to do that you can run the following command.

The above command will also map the certs for each and every Fabric CA. This must help you in the case of regeneration of certs.

Step 4 : Deploy the containers

Finally, all we have to do is to deploy the network to the docker swarm. I’ve created a single script for it.

This command will run a set of commands, each will deploy the services from the compose files to the targeted node.

To give more clarity, we’re currently deploying the services/containers from our manager node to all the targeted nodes in the swarm network. Which means you don’t have to run any other script other that docker join and move crypto in other nodes in the network. Which makes the process quite easier.

Usually it’ll take some time based on the network speed and the machine’s performance. Once it’s been completed, in order, to make sure everything is fine, just run the following command and make sure nothing is returned.

docker service ls | grep "0/1"

The above command basically checks if there is any failed containers. If in case you find any failed containers, run the following commands to debug what went wrong.

docker service ps <service id>

I’ve listed the most common errors/challenges you’ll face in this setup down below. Please have a look at it in case if you are stuck somewhere.

Step 5 : Setting Up Channels & Chaincode

And in order to complete this we need to run the channels creations, joining and install and instantiating of chaincodes. This is more similar to the procedure we follow in single machine deployment.

In our sample, just run following commands to achieve them.

./scripts/create_channel.sh ./scripts/install_chaincodes.sh

This will make sure of Fabric CLI service to setup channels & chaincodes.

Once everything is done, you’ll have a multi host hyperledger fabric network. If you want to set up for more number of nodes, you just have to create more compose files, and corresponding organization configuration.

Common Errors & Mistakes

This following list is the curated items that one might face during this setup and it’s resolution.

  1. If you face “Invalid mount config” in you docker service logs, please check if you have the certs in all the machines at right location.
  2. CA Failed? It’s mostly because of the invalid certs provided to it. Try running populate_hostname.sh again and see.
  3. If you face network issues, make sure you ran create_network.sh before start_all.sh

This boilerplate is been used by various of our clients and also a lot of our production environments.


This is a companion discussion topic for the original entry at https://www.skcript.com/svr/setting-up-a-multi-node-hyperledger-fabric-network-with-docker-swarm/

(Abiram) #4

Hi Team,

I am new to the world of deploying the HL Fabric application in a production environment.

I am trying to deploy my application with three AWS EC2 Ubuntu 18.04 version instance using this scripts just wanted to check as inside the **

\hlf-docker-swarm\network\kafka-zookeeper

The configuration for starting the Kafka-Zookeeper orderer consensus is mentioned only once.

Does this means that kafka-Zookepper will be setup on just on one Node Machine be it ORG1 , ORG2 or ORG3 ?

Because as per the configuration we are setting up the orderer instance on all the 3 nodes .
**


(Varun Raj) #5

Hello @abhi47roy,

Good to know that :slight_smile: Actually you can change the configuration to run it on any nodes as well as distribute to three nodes. In my article I’ve made it to work in single node to give simplicity :slight_smile:


(Abiram) #6

Thanks Varun for the fast reply.

So for testing that HL Fabric blockchain application is working using this repository we can proceed with single kafka-Zookepper instance.

placement:
** constraints:**
** - node.hostname == intel-sawtooth**

The above configuration that is getting used for the deploying the network using docker stack deploy we need to change the “intel-sawtooth” with the hostname of our AWS EC2 instance machine in all of the docker-compose files to make it up and running.


(Varun Raj) #7

Yes that right, just change the hostname to any instance’s hostname that have joined in your swarm network.


(Abiram) #8

Well after making minimum changes i was able to make all the container up on all the leader machine and worker machine. After joining all the peers to the channel named as “mychannel” and instantiating the chaincode .

when i did “docker ps -a” on all the machine in the docker swarm , the container for the chaincode was only created on the leader machine not the worker machine.

How can I verify whether container has been created for org2 and org3 related to the chaincode that is installed and instantiated.

Or is there any kind of thing like docker stack deploy chaincode


(Varun Raj) #9

When you instantiate the chain-code, only one chain-code container will be created which is for the peer where you instantiated. But when you do the first transaction endorsement in the other peers it gets created automatically


(Abiram) #10

So I was able to do a invoke and query transaction successfully on the peer on which the cli services are present in our case org1 and it resulted with the desired value but when i went back to the worker machine and checked if the containers for the chaincode got generated or not . But no luck. :roll_eyes:


(Saeedi) #11

If containers for peers, orderer and ca are running on worker node and you are having trouble executing invoke or query command. In manager node, script uses following command to do a query

docker exec hlf_services_cli.1.l5iv1ts55wdgh6idxibhxdboq peer chaincode query …

As you can see it uses or it first access cli container present on manager node and there is no cli container running on worker node. So for same command to work on worker node, use docker ps on worker node and copy the name of peer0 container and replace it with that in above command. i.e. above command would become

docker exec hlf_peer_peer0_org2.1.o2xcs7xb7hjo4fe2sc3ke62li peer chaincode query …

It worked in my case. Hope it works for you too.


(Abiram) #12

Hi Saeedi,

Thanks for the reply . it helped .
The scenario you told will be then only working for the query commands.

But when I am trying to do a invoke from the worker node it is throwing error as :

Error: Error getting broadcast client: failed to load config for OrdererClient: unable to load orderer.tls.rootcert.file: open /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer0.example.com/msp/tlscacerts/tlsca.example.com-cert.pem: no such file or directory

So the command I used for invoke on org3 is :

docker exec hlf_peer_peer0_org3.1.lkyv7vijri5lvbueiyvuv0ia6 peer chaincode invoke -o orderer2.example.com:7050 --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer2.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -C mychannel -n simplecc2 -c '{“Args”:[“open”,“104”,“2000”]}'

but no success even though all the certs related to the orderer are present in the above mentioned path that is /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer2.example.com/msp/tlscacerts/tlsca.example.com-cert.pem.

Is there anything in the configuration file that i am missing ?


(Saeedi) #13

Try replacing orderer2 with orderer0 in command. Just for testing purpose. If you are using multiple machines or vms, make sure that you have executed move_crypto.sh on every other machine.


(Sri Harsha) #14

OrdererOrgs:

  • Name: Orderer
    Domain: example.com
    Specs:
    • Hostname: orderer
      Template:
      Count: 4
      Specs:
    • Hostname: orderer0
    • Hostname: orderer1
    • Hostname: orderer2
    • Hostname: orderer3
    • Hostname: orderer4
    • Hostname: orderer5
    • Hostname: orderer6

Wouldn’t mentioning specs and templates cause a naming conflict?
From fabric docs:
Template and Specs are not mutually exclusive. You may define both
sections and the aggregate nodes will be created for you. Take care with name collisions.