cess-indexer

Introduction

indexer is a simple cacher search program. It only needs simple configuration to run the program to obtain services. Please run the program in the Linux operating system. Before doing so, please make sure that the Golang environment is installed on your machine.

Start

1. First, you need to make a simple configuration. The configuration file is config.toml under the config directory,Please fill in all configuration options.

#CESS network ws address
RpcAddr="wss://devnet-rpc.cess.cloud/ws/"
#You indexer server port
ServerPort="8081"
#You CESS account and seed
AccountSeed="plug increase image genius rabbit unable once sponsor tail castle wet fan"
AccountID="cXgP2nH3GkFweiAUcPFWJjxRdiScooTneMgdowKe6Lfe3hQQ8"
#This configuration is used to adjust the size of the transaction pool
MaxTxPoolSize=128
#order expiration(hours)
ExpDate=24

2. Then you can go back to the project main directory and use the command go run main.go to start the service

Unit Test

You can use the test samples in the test directory for unit testing. Note that you should set the configuration file before testing

cd test
go test indexer_test.go

Code Walkthrough

1. The user calls the CreateCacheBill service through the http protocol to generate a new cache order.Then the data will be stored in the cache pool. When the cache pool is full or reaches the predetermined time, it will be submitted to the blockchain together.

   cess-indexer/server/service/index.go

   Lines 17 to 39 in 87325c6

   	func CreateCacheBill(fhash, shash string) (map[string]string, resp.Error) {
   	resMap := make(map[string]string)
   	fmeta, err := chain.GetChainCli().GetFileMetaInfo(fhash)
   	if err != nil {
   	return resMap, resp.NewError(500, errors.Wrap(err, "create cache bill error"))
   	}
   	var size uint64
   	if isInvalidFileSlice(shash, fmeta, &size) {
   	err = errors.New("bad file slice hash")
   	return resMap, resp.NewError(400, errors.Wrap(err, "create cache bill error"))
   	}
   	bid, cacher, err := index.CreateAndSendCacheBill(
   	fhash, shash, uint64(size),
   	time.Duration(config.GetConfig().ExpDate)*time.Hour,
   	)
   	if err != nil {
   	return resMap, resp.NewError(400, errors.Wrap(err, "create cache bill error"))
   	}
   	resMap["BID"] = hex.EncodeToString(bid[:])
   	resMap["cacherAddr"] = cacher
   	resMap["tips"] = "please get token after about 15 s"
   	return resMap, nil
   	}

2. The user can also call the CreateCacheBillBySliceIndex method to create a cache order. The difference from the former is that the file partition index is used instead of the hash value to complete the location.

   cess-indexer/server/service/index.go

   Lines 41 to 62 in 87325c6

   	func CreateCacheBillBySliceIndex(fhash string, i int) (map[string]string, resp.Error) {
   	resMap := make(map[string]string)
   	fmeta, err := chain.GetChainCli().GetFileMetaInfo(fhash)
   	if err != nil {
   	return resMap, resp.NewError(500, errors.Wrap(err, "create cache bill error"))
   	}
   	if i >= len(fmeta.BlockInfo) || i < 0 {
   	err = errors.New("invalid file slice index")
   	return resMap, resp.NewError(400, errors.Wrap(err, "create cache bill error"))
   	}
   	bid, cacher, err := index.CreateAndSendCacheBill(
   	fhash, string(fmeta.BlockInfo[i].BlockId[:]), uint64(fmeta.BlockInfo[i].BlockSize),
   	time.Duration(config.GetConfig().ExpDate)*time.Hour,
   	)
   	if err != nil {
   	return resMap, resp.NewError(400, errors.Wrap(err, "create cache bill error"))
   	}
   	resMap["BID"] = hex.EncodeToString(bid[:])
   	resMap["cacherAddr"] = cacher
   	resMap["tips"] = "please get token after about 15 s"
   	return resMap, nil
   	}

   The two services are called through the same URL. The only difference is that you choose to use the hash or sine parameter.When creating an bill, all active miners on the blockchain will be queried, and the miners who have cached the file fragments will be preferentially selected to provide download services. If there are multiple miners, the miners with the best unit price and distance will be selected first.

   cess-indexer/base/index/index.go

   Lines 269 to 284 in 87325c6

   	func CreateAndSendCacheBill(fhash, shash string, size uint64, exp time.Duration) (BID, string, error) {
   	cachers, err := client.QueryMinersCachedFile(fhash, GetCachers())
   	if err != nil {
   	return BID{}, "", errors.Wrap(err, "create and send cache bill error")
   	}
   	if len(cachers) <= 0 {
   	cachers = getAllCaches()
   	}
   	cacher := GetBestCacher(cachers...)
   	bill, err := CreateCacheBill(cacher, fhash, shash, size, exp)
   	if err != nil {
   	return BID{}, "", errors.Wrap(err, "create and send cache bill error")
   	}
   	GetOrders().Put(bill)
   	return bill.Id, cacher, nil
   	}

3. After the cache bill is created, users can call the GenerateFileToken service through HTTP request to obtain the download link according to the response data obtained previously.

   cess-indexer/server/service/index.go

   Lines 64 to 89 in 87325c6

   	func GenerateFileToken(cacher, hexBid string) (map[string]string, resp.Error) {
   	resMap := make(map[string]string)
   	b, err := hex.DecodeString(hexBid)
   	if err != nil {
   	return resMap, resp.NewError(400, errors.Wrap(err, "generate file token error"))
   	}
   	bid := index.BID{}
   	copy(bid[:], b)
   	hash, ok := index.GetOrders().GetTxHash(bid)
   	if !ok {
   	resMap["tips"] = "the bill is not ready"
   	return resMap, resp.NewError(0, nil)
   	}
   	kr, _ := keyring.FromURI(config.GetConfig().AccountSeed, keyring.NetSubstrate{})
   	// sign message
   	sign, err := kr.Sign(kr.SigningContext([]byte(hash + hexBid)))
   	if err != nil {
   	return resMap, resp.NewError(400, errors.Wrap(err, "generate file token error"))
   	}
   	token, err := client.GenerateToken(cacher, hash, hexBid, sign[:])
   	if err != nil {
   	return resMap, resp.NewError(400, errors.Wrap(err, "generate file token error"))
   	}
   	resMap["url"] = "http://" + cacher + "/download/file/" + token
   	return resMap, nil
   	}

   At this time, the indexer will send a request to the cache to generate a token. After receiving the response data, the indexer will splice the cache file download address and the token into a download link and return.

   cess-indexer/base/client/client.go

   Lines 26 to 45 in 87325c6

   	func GenerateToken(addr, hash, bid string, sign []byte) (string, error) {
   	var token string
   	u, err := url.JoinPath("http://"+addr, "auth", "gen")
   	if err != nil {
   	return token, errors.Wrap(err, "generate token error")
   	}
   	headers := Pairs{"Content-Type": "application/json"}
   	jbytes, err := json.Marshal(AuthReq{hash, bid, sign})
   	if err != nil {
   	return token, errors.Wrap(err, "generate token error")
   	}
   	resp, err := HttpRequest("POST", u, headers, strings.NewReader(string(jbytes)))
   	if err != nil {
   	return token, errors.Wrap(err, "generate token error")
   	}
   	if PraseData(&token, resp.Data) != nil {
   	return token, errors.Wrap(err, "generate token error")
   	}
   	return token, nil
   	}

API

There are currently two APIs available

Create cache bill

This API is used to generate cache bill. The indexer will automatically select the nearest and cheapest cacher to provide you with cache services,The use method is as follows:

POST /order/bill/create

• Request Header

key	value
Content-Type	application/json

• Request Body

field	value
fid	file hash in CESS storage
sname	file slice hash,is optional,you can choose to use sname or sindex
sindex	index of file slice ,startiing from 0,

• Responses

Response schema: application/json

HTTP Code	Message	Description
200	message map	return Bill ID,cacher address and tips
400	error	specific error reminder

Generate token

This API is used to generate cache file download token. Indexer will return a download link with token for you. You can use this link to download files from cacher on any device.This API needs to use the bill info generated in the previous API.

POST /order/token/gen

• Request Header

key	value
Content-Type	application/json

• Request Body

field	value
cacher	cacher address,please obtain it from 'Create cache bill' API response
BID	bill ID,please obtain it from 'Create cache bill' API response

• Responses

Response schema: application/json

HTTP Code	Message	Description
200	message map	return cache download url or tips
400	error	specific error reminder