Running Hyperledger Composer with multiple organization on different host machines


(Naveen Honest Raj) #1

I deployed a single organization model in Hyperledger Composer successfully.
Now, I am trying to deploy two organization network in two different host machine. But I am facing some errors related to TLS. I tried enabling it and now I am facing “Handshake errors”. Anyone successfully deployed it without any errors?


(Varun Raj) #2

For multiple organization, you need to enable TLS options in all the places.

For peer you need to add the following environment variables.

- CORE_PEER_TLS_ENABLED=true
- CORE_PEER_TLS_CERT_FILE=/etc/hyperledger/peer/tls/server.crt
- CORE_PEER_TLS_KEY_FILE=/etc/hyperledger/peer/tls/server.key
- CORE_PEER_TLS_ROOTCERT_FILE=/etc/hyperledger/peer/tls/ca.crt

And in the CA and Orderer

- FABRIC_CA_SERVER_TLS_ENABLED=true
- FABRIC_CA_SERVER_TLS_CERTFILE=/etc/hyperledger/fabric-ca-server-config/ca.org2.trucerts.com-cert.pem
- FABRIC_CA_SERVER_TLS_KEYFILE=/etc/hyperledger/fabric-ca-server-config/5f514f7e164745ab7458ac2a2fdd705981e32333709318c41f03226fb176838f_sk
- ORDERER_GENERAL_TLS_ENABLED=true
- ORDERER_GENERAL_TLS_PRIVATEKEY=/etc/hyperledger/msp/orderer/tls/server.key
- ORDERER_GENERAL_TLS_CERTIFICATE=/etc/hyperledger/msp/orderer/tls/server.crt
- ORDERER_GENERAL_TLS_ROOTCAS=[/etc/hyperledger/msp/orderer/tls/ca.crt]

Make sure that you’ve mounted the corresponding volumes.

And then you should also enable the tls options while creating and joining the channel.


(VK Blue) #3
  • Have all environment variables and path defined correctly?
  • Use IP address, rather than using localhost.
  • Ping the hosts from one to the another, vice versa.
  • Add correct host file entries, specially in case of orderer.
  • Enable TLS.
  • Enable and configure CA correctly.
  • Fabric environment should be up and please ensure you have the sample chaincode deployed. This eliminates all silly mistakes we do when we configure peers and channels
  • Endorement policy should be defined according to the consortium definition
  • Connection json files for Composer should be 100% correct.

Hope this helps.


#4

I am trying to deploy two peers in two different hosts machines but i got the Error: Error trying login and get user Context. Error: Error trying to enroll user or load channel configuration. Error: Calling enrollment endpoint failed with error [Error: connect ECONNREFUSED 127.0.0.1:7054]
Command failed

I have enabled my TLS options and the above environment variables correctly ,
I have also checked logs for cli where i got a warning : 2018-01-19 09:08:30.433 UTC [nodeCmd] createChaincodeServer -> WARN 001 peer.chaincodeListenAddress is not set, use peer.listenAddress 0.0.0.0:7051

and have also checked logs in peer container :

2018-01-19 09:17:16.932 UTC [nodeCmd] serve -> INFO 001 Starting peer:
 Version: 1.0.4
 Go version: go1.7.5
 OS/Arch: linux/amd64
 Chaincode:
  Base Image Version: 0.3.2
  Base Docker Namespace: hyperledger
  Base Docker Label: org.hyperledger.fabric
  Docker Namespace: hyperledger

2018-01-19 09:17:16.932 UTC [ledgermgmt] initialize -> INFO 002 Initializing ledger mgmt
2018-01-19 09:17:16.932 UTC [kvledger] NewProvider -> INFO 003 Initializing ledger provider
2018-01-19 09:17:16.933 UTC [kvledger.util] CreateDirIfMissing -> DEBU 004 CreateDirIfMissing [/var/hyperledger/production/ledgersData/ledgerProvider/]
2018-01-19 09:17:16.933 UTC [kvledger.util] logDirStatus -> DEBU 005 Before creating dir - [/var/hyperledger/production/ledgersData/ledgerProvider/] exists
2018-01-19 09:17:16.933 UTC [kvledger.util] logDirStatus -> DEBU 006 After creating dir - [/var/hyperledger/production/ledgersData/ledgerProvider/] exists
panic: Error while trying to open DB: resource temporarily unavailable

goroutine 1 [running]:
panic(0xc70020, 0xc42034fc20)
        /opt/go/src/runtime/panic.go:500 +0x1a1
github.com/hyperledger/fabric/common/ledger/util/leveldbhelper.(*DB).Open(0xc420342b80)
        /opt/gopath/src/github.com/hyperledger/fabric/common/ledger/util/leveldbhelper/leveldb_helper.go:88 +0x237
github.com/hyperledger/fabric/core/ledger/kvledger.openIDStore(0xc420342b40, 0x36, 0x1)
        /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/kvledger/kv_ledger_provider.go:267 +0x16f
github.com/hyperledger/fabric/core/ledger/kvledger.NewProvider(0xcb7d20, 0x0, 0xc4203526c0, 0x0)
        /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/kvledger/kv_ledger_provider.go:67 +0xf7
github.com/hyperledger/fabric/core/ledger/ledgermgmt.initialize()
        /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/ledgermgmt/ledger_mgmt.go:59 +0x14e
github.com/hyperledger/fabric/core/ledger/ledgermgmt.Initialize.func1()
        /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/ledgermgmt/ledger_mgmt.go:49 +0x14
sync.(*Once).Do(0x145adf0, 0xe3f380)
        /opt/go/src/sync/once.go:44 +0xdb
github.com/hyperledger/fabric/core/ledger/ledgermgmt.Initialize()
        /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/ledgermgmt/ledger_mgmt.go:50 +0x39
github.com/hyperledger/fabric/peer/node.serve(0xc42034ec00, 0x0, 0x1, 0x0, 0x0)
        /opt/gopath/src/github.com/hyperledger/fabric/peer/node/start.go:88 +0x113
github.com/hyperledger/fabric/peer/node.glob..func1(0x14016e0, 0xc42034ec00, 0x0, 0x1, 0x0, 0x0)
        /opt/gopath/src/github.com/hyperledger/fabric/peer/node/start.go:75 +0x3f
github.com/hyperledger/fabric/vendor/github.com/spf13/cobra.(*Command).execute(0x14016e0, 0xc42034ebe0, 0x1, 0x1, 0x14016e0, 0xc42034ebe0)
        /opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/spf13/cobra/command.go:599 +0x234
github.com/hyperledger/fabric/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x1401d40, 0xf, 0xc42000e075, 0x6)
        /opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/spf13/cobra/command.go:689 +0x367
github.com/hyperledger/fabric/vendor/github.com/spf13/cobra.(*Command).Execute(0x1401d40, 0x1b, 0xc42000e075)
        /opt/gopath/src/github.com/hyperledger/fabric/vendor/github.com/spf13/cobra/command.go:648 +0x2b
main.main()
        /opt/gopath/src/github.com/hyperledger/fabric/peer/main.go:118 +0x54e

I don’t know where to look into. any suggestions??


(Naveen Honest Raj) #5

You might need to update your /etc/hosts file temporarily to make it ping the second machine. This will temporarily help you to make sure whether there is no error in connecting to one peer to another peer/orderer.

192.168.xx.xx orderer.example.com
192.168.xx.xx peer0.org1.example.com
192.168.xx.xx peer0.org2.example.com
192.168.xx.xx ca.org1.example.com
192.168.xx.xx ca.org2.example.com


#6

I have already did that . did you used hyperledger fabric or Composer for multi org in diff hosts ?


(Varun Raj) #7

Hi @abhi119

Hyperledger Composer is an chaincode building framework and it runs on top of hyperledger fabric, thus you need both to setup a successful network.


#8

@varun @navdevl can You give me the end to end steps with some sample working code so i can check if i have missed something?


#9

@varun I have used fabric to join two peers of two different host machines in a single channel and later i have used composer to create and deploy the BNA file.


(Naveen Honest Raj) #10

@abhi119 Are you able to run a single organization network successfully? If so, IMO first try moving only the orderer to a different machine and then see whether the network still works flawlessly. Then you can understand the workflow for hosting multiorganization in multiple machines.


#11

@navdevl yes i did run a single organization network successfully as well as multiple organization in a single machine . I can create orderer for multiple hosts . Till now i have joined two peers of different hosts in a single channel after that i put my composer files (sample-network) in fabric and try to deploy the bna file . but got ECDSA error while running runtime install command:Error: Error trying install composer runtime. Error: No valid responses from any peers.
Response from attempted peer comms was an error: Error: Failed to deserialize cr eator identity, err The supplied identity is not valid, Verify() returned x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verifi cation failure" while trying to verify candidate authority certificate "ca.polic e.gov.uk")

Command failed
note: no errors in the docker CA containers
I don’t know if i am in the right direction . any help will be appreciated


(Naveen Honest Raj) #12

@abhi119 I am not sure this will help you. But just give it a try.
Try docker images list and see if there are any dev-peer-* images. If so try deleting them using docker rmi -f container_ids.

ECDSA failure sometimes happens because the chaincode image would have been created already with different CA-certs and failed to replaced with new one. Try this and let me know. I can share my knowledge gained by makings lots of mistakes :smiley:


#13
CONTAINER ID        IMAGE                                     COMMAND                  CREATED             STATUS              PORTS                                            NAMES
b66bf7b4320a        hyperledger/fabric-orderer                "/bin/bash -c 'cat /…"   10 minutes ago      Up 10 minutes       0.0.0.0:7050->7050/tcp                           orderer.gov.uk
f578302c7fd4        hyperledger/fabric-ca:x86_64-1.0.4        "sh -c 'fabric-ca-se…"   11 minutes ago      Up 11 minutes       0.0.0.0:7054->7054/tcp                           ca.police.gov.uk
b37163e9e72e        hyperledger/fabric-tools                  "/bin/bash"              11 minutes ago      Up 11 minutes                                                        cli
ea737c1db8e8        hyperledger/fabric-peer                   "/bin/bash -c 'cat /…"   11 minutes ago      Up 11 minutes       0.0.0.0:7051->7051/tcp, 0.0.0.0:7053->7053/tcp   peer0.police.govuk
2fed445e450d        hyperledger/fabric-couchdb:x86_64-1.0.4   "tini -- /docker-ent…"   11 minutes ago      Up 11 minutes       4369/tcp, 9100/tcp, 0.0.0.0:5984->5984/tcp       couchdb

#14

@navdevl I have no such containers or images dev-peer-* . above are the containers i have created


(Naveen Honest Raj) #15

@abhi119 docker ps and docker images are different commands. I wanted you to see whether there are images created. Not containers.


#16

@navdevl Yes i know but there is no images shown on docker images list command

/conf$ docker images list
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

for docker images command its shows :
 conf$ docker images
REPOSITORY                     TAG                 IMAGE ID            CREATED             SIZE
hyperledger/fabric-ca          latest              8e691b3509bf        2 months ago        238MB
hyperledger/fabric-ca          x86_64-1.0.4        8e691b3509bf        2 months ago        238MB
hyperledger/fabric-tools       latest              6051774928a6        2 months ago        1.33GB
hyperledger/fabric-tools       x86_64-1.0.4        6051774928a6        2 months ago        1.33GB
hyperledger/fabric-couchdb     latest              cf24b91dfeb1        2 months ago        1.5GB
hyperledger/fabric-couchdb     x86_64-1.0.4        cf24b91dfeb1        2 months ago        1.5GB
hyperledger/fabric-kafka       latest              7a9d6f3c4a7c        2 months ago        1.29GB
hyperledger/fabric-kafka       x86_64-1.0.4        7a9d6f3c4a7c        2 months ago        1.29GB
hyperledger/fabric-zookeeper   latest              53c4a0d95fd4        2 months ago        1.3GB
hyperledger/fabric-zookeeper   x86_64-1.0.4        53c4a0d95fd4        2 months ago        1.3GB
hyperledger/fabric-orderer     latest              b17741e7b036        2 months ago        151MB
hyperledger/fabric-orderer     x86_64-1.0.4        b17741e7b036        2 months ago        151MB
hyperledger/fabric-peer        latest              1ce935adc397        2 months ago        154MB
hyperledger/fabric-peer        x86_64-1.0.4        1ce935adc397        2 months ago        154MB
hyperledger/fabric-javaenv     latest              a517b70135c7        2 months ago        1.41GB
hyperledger/fabric-javaenv     x86_64-1.0.4        a517b70135c7        2 months ago        1.41GB
hyperledger/fabric-ccenv       latest              856061b1fed7        2 months ago        1.28GB
hyperledger/fabric-ccenv       x86_64-1.0.4        856061b1fed7        2 months ago        1.28GB
hyperledger/fabric-baseos      x86_64-0.4.2        6be1916d236d        4 months ago        130MB
hyperledger/fabric-baseos      x86_64-0.3.2        bbcbb9da2d83        5 months ago        129MB

(Naveen Honest Raj) #17

These are the things you could do to find where things went wrong.

  1. Try automating the process of replacing *_sk file (TLS Certs) in the docker-compose file by giving it as an environment variable which you can find by $(ls blah/blah/*_sk). In this way, you will avoid making mistake in this way. This is where many people make mistakes

  2. Make sure whether the hosts file is configured properly to communicate from one physical-machine to another-physical machine. Either configure them in your docker-compose files or configure them by adding entries in /etc/hosts file.

  3. Run logs in all the docker containers. Peers, CA, and orderer. This will help you figure out where the process-fails ultimately.]

If all these didn’t help you, try answering these questions.

  1. Did BYFN-on-multiple machines worked?
  2. Did you really automated the copy-paste-replace things.?
  3. LOGS didn’t really show you anything other than ECDSA failure?

#18

@navdevl what are the versions of fabric and composer did you used my fabric version is 1.04 and composer version is 16.3 in link https://github.com/hyperledger/composer/releases its given composer v16.3 is compatible with fabric v1.0


(Naveen Honest Raj) #19

@abhi119 Yes. I had it running successfully with following configurations.

  • Fabric 1.04 with Composer 16.x

  • Fabric 1.1-preview with Composer 17.x

Both these combination of configuration worked fine.

What is the problem you are facing currently?


#20

@navdevl I am getting error ⠹

⠙ Installing runtime for business network sample-network. This may take a minute...E0201 07:39:55.309004271   22516 ssl_transport_security.c:921] Handshake failed with fatal error SSL_ERROR_SSL: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number.
✖ Installing runtime for business network sample-network. This may take a minute...
Error: Error trying install composer runtime. Error: No valid responses from any peers.
Response from attempted peer comms was an error: Error: Connect Failed

I have disabled my TLS except for orderer . I have checked the logs and it looks both machines detect each other