Issue with operation with allow bearer token for object operations (object not found)

[anatoly-bogatyrev]

NeoFS version 0.19.0-42-gb587b23 (latest master)

The issue with GET operation with allow bearer token.
Detailed log: https://drive.google.com/file/d/1Zv1GoAfeieWfmmUH3rcGi8s9C5AUXv9H/view?usp=sharing
Can be opened after download as .HTML file.

[alexvanin]

Provide used bearer token and eACL.

[anatoly-bogatyrev]

$ cat TemporaryDir/bearer_allow_all_user
{"body":{"eaclTable":{"version":{"major":2, "minor":6}, "containerID":{"value":"9xhFSrBPyAaJDHHTaNb1cDdq5eACFNLkXf6mVjlrkYA="}, "records":[{"operation":"GET", "action":"ALLOW", "filters":[], "targets":[{"role":"USER", "keys":[]}]}, {"operation":"HEAD", "action":"ALLOW", "filters":[], "targets":[{"role":"USER", "keys":[]}]}, {"operation":"PUT", "action":"ALLOW", "filters":[], "targets":[{"role":"USER", "keys":[]}]}, {"operation":"DELETE", "action":"ALLOW", "filters":[], "targets":[{"role":"USER", "keys":[]}]}, {"operation":"SEARCH", "action":"ALLOW", "filters":[], "targets":[{"role":"USER", "keys":[]}]}, {"operation":"GETRANGE", "action":"ALLOW", "filters":[], "targets":[{"role":"USER", "keys":[]}]}, {"operation":"GETRANGEHASH", "action":"ALLOW", "filters":[], "targets":[{"role":"USER", "keys":[]}]}, {"operation":"GET", "action":"DENY", "filters":[], "targets":[{"role":"USER", "keys":[]}]}, {"operation":"HEAD", "action":"DENY", "filters":[], "targets":[{"role":"USER", "keys":[]}]}, {"operation":"PUT", "action":"DENY", "filters":[], "targets":[{"role":"USER", "keys":[]}]}, {"operation":"DELETE", "action":"DENY", "filters":[], "targets":[{"role":"USER", "keys":[]}]}, {"operation":"SEARCH", "action":"DENY", "filters":[], "targets":[{"role":"USER", "keys":[]}]}, {"operation":"GETRANGE", "action":"DENY", "filters":[], "targets":[{"role":"USER", "keys":[]}]}, {"operation":"GETRANGEHASH", "action":"DENY", "filters":[], "targets":[{"role":"USER", "keys":[]}]}]}, "ownerID":null, "lifetime":{"exp":"100500", "nbf":"1", "iat":"0"}}, "signature":{"key":"A0Zqhwyc1HtPSGPj0CVlJZ01T5e4pDpKwfu7cRU45ZGc", "signature":"BKLFvhKRtuXB0aq3TSbVK4P8NYLoAL+KYcvdl97DqfjQVlmpOXBU3rGhFHDXVRL72kxsCrQuvO1STAu8/KqKoz4="}}

[anatoly-bogatyrev]

Next run: https://drive.google.com/file/d/1RL8C2C9_DmuwY1a1cudC4vDZalPmPBOe/view?usp=sharing
in the first and in the second cases, the token passes successfully - the error of the operation in the output is not in the incorrect permissions to the operation, but in the fact that the object has not been found. In the first case on the GET, in the second case on the Search operation.

[anatoly-bogatyrev]

How to reproduce:

1. Init the public container:

21:45:06.967	INFO	Cmd: neofs-cli --rpc-endpoint s01.neofs.devenv:8080 --key L3fjPcA8kJUUYPQ8cxL6uQaHQfaR5ARkXwCbba1xHrdxz1SV1VjC container create --policy "REP 2 IN X CBF 1 SELECT 2 FROM * AS X" --basic-acl 0x0FFFFFFF --await	
21:45:09.015	INFO	Output: container ID: 5o5k41YnBmrMqrvB2iY3HTSAjpQ7TCnvnhFpwAvbiRkA
awaiting...
container has been persisted on sidechain
21:45:09.016	INFO	Created container 5o5k41YnBmrMqrvB2iY3HTSAjpQ7TCnvnhFpwAvbiRkA with rule 'REP 2 IN X CBF 1 SELECT 2 FROM * AS X'	
21:45:09.017	INFO	${PUBLIC_CID_GEN} = 5o5k41YnBmrMqrvB2iY3HTSAjpQ7TCnvnhFpwAvbiRkA

2. PUT object for 70Mb:

21:45:09.025	INFO	Going to put the object	
21:45:09.025	INFO	Storage nodes: ['s01.neofs.devenv:8080', 's02.neofs.devenv:8080', 's03.neofs.devenv:8080', 's04.neofs.devenv:8080']	
21:45:09.025	INFO	Cmd: neofs-cli --rpc-endpoint s02.neofs.devenv:8080 --key L3fjPcA8kJUUYPQ8cxL6uQaHQfaR5ARkXwCbba1xHrdxz1SV1VjC object put --file 150d543b-7162-4b82-8190-e67b0a84f499 --cid 5o5k41YnBmrMqrvB2iY3HTSAjpQ7TCnvnhFpwAvbiRkA  --attributes key1=1,key2=abc 	
21:45:12.792	INFO	Output: [150d543b-7162-4b82-8190-e67b0a84f499] Object successfully stored
  ID: 2mDiGsVyUzVRagsvNFAvGhxVN1vSpA7DuXXBydSnzMdz
  CID: 5o5k41YnBmrMqrvB2iY3HTSAjpQ7TCnvnhFpwAvbiRkA
21:45:12.794	INFO	${S_OID_USER} = 2mDiGsVyUzVRagsvNFAvGhxVN1vSpA7DuXXBydSnzMdz

3. Set eACL to DENY all operation for USER

Start / End / Elapsed:	20210520 21:45:24.414 / 20210520 21:45:25.476 / 00:00:01.062
21:45:24.415	INFO	Cmd: neofs-cli --rpc-endpoint s01.neofs.devenv:8080 --key L3fjPcA8kJUUYPQ8cxL6uQaHQfaR5ARkXwCbba1xHrdxz1SV1VjC container set-eacl --cid 5o5k41YnBmrMqrvB2iY3HTSAjpQ7TCnvnhFpwAvbiRkA --table TemporaryDir/gen_eacl_deny_all_USER --await	
21:45:25.474	INFO	Output: awaiting...
EACL has been persisted on sidechain

4. Sleep 30sec
5. Check that request will be failed as expected (correct behavior):

21:45:55.577	INFO	Cmd: neofs-cli --rpc-endpoint s03.neofs.devenv:8080 --key L3fjPcA8kJUUYPQ8cxL6uQaHQfaR5ARkXwCbba1xHrdxz1SV1VjC object put --file 150d543b-7162-4b82-8190-e67b0a84f499 --cid 5o5k41YnBmrMqrvB2iY3HTSAjpQ7TCnvnhFpwAvbiRkA  --attributes key1=1,key2=abc 	
21:45:55.626	FAIL	command 'neofs-cli --rpc-endpoint s03.neofs.devenv:8080 --key L3fjPcA8kJUUYPQ8cxL6uQaHQfaR5ARkXwCbba1xHrdxz1SV1VjC object put --file 150d543b-7162-4b82-8190-e67b0a84f499 --cid 5o5k41YnBmrMqrvB2iY3HTSAjpQ7TCnvnhFpwAvbiRkA  --attributes key1=1,key2=abc ' return with error (code 1): Error: can't put object: closing the stream failed: rpc error: code = Unknown desc = access to operation 3 is denied by extended ACL check

6. Now try to execute operations with bearer token (any)

21:46:00.687	INFO	Cmd: neofs-cli --rpc-endpoint s01.neofs.devenv:8080 --key L3fjPcA8kJUUYPQ8cxL6uQaHQfaR5ARkXwCbba1xHrdxz1SV1VjC object get --cid 5o5k41YnBmrMqrvB2iY3HTSAjpQ7TCnvnhFpwAvbiRkA --oid 2mDiGsVyUzVRagsvNFAvGhxVN1vSpA7DuXXBydSnzMdz --file TemporaryDir/local_file_eacl --bearer TemporaryDir/bearer_allow_all_user --xhdr a=256	
21:46:00.755	FAIL	command 'neofs-cli --rpc-endpoint s01.neofs.devenv:8080 --key L3fjPcA8kJUUYPQ8cxL6uQaHQfaR5ARkXwCbba1xHrdxz1SV1VjC object get --cid 5o5k41YnBmrMqrvB2iY3HTSAjpQ7TCnvnhFpwAvbiRkA --oid 2mDiGsVyUzVRagsvNFAvGhxVN1vSpA7DuXXBydSnzMdz --file TemporaryDir/local_file_eacl --bearer TemporaryDir/bearer_allow_all_user --xhdr a=256' return with error (code 1): Error: can't put object: reading the response failed: rpc error: code = Unknown desc = object not found

Instead of getting an object or even an access error (if the token would be incorrect) - the object was not found

[alexvanin]

This bug happens only when storage node re-sends requests. For example if you try to make object.Head request to the node that stores object locally, you'll get correct response.

When request re-sended, remote storage nodes can't verify extended ACL. We don't inherit errors in NeoFS right now, thus in the end user gets object was not found even though remote storage nodes returned `access to operation 2 is denied by extended ACL check". Future NeoFS error design discussed there.

Remote storage nodes can't verify extended ACL because they are not traversing linked list of request meta headers, but they should.