Fix sync with new native clients

[BlackDex]

This should fix syncing with the new native clients.

• Filter out password history items which have null
• Redesigned the Secure Note Type check to be more robust
• Check for null passwords in history during import (or key rotation)
  This should not be an issue during the key rotation, since we do not return the invalid password history items anymore.

Fixes #4870
Fixes #4921

[BJReplay]

I had hoped this would fix an issue that I had with a vault that wouldn't sync both with the old and the new client, which I suspected was due to corrupted data - the sync would be requested, but would fail, but I tried it on my recovery machine (where I restore a backup every day) and it didn't help when I changed my image from latest to testing. The dodgy account won't sync until I delete all of the items from trash. There are one or more items in trash that prevent syncing both on old and new clients, both with :latest and :testing docker images.

With around 600 items in trash, it's a relatively big haystack to find the cipher that is preventing syncing, but I'll see if i can get a chance to do so, to find out why it is causing the issue.

[BlackDex]

@BJReplay if it also prevents syncing on an older client that is weird. But would be good to know what would be breaking this indeed.

[BJReplay]

@BlackDex I have backups of the corrupted data that I can restore into a test environment. I am travelling for a few days, but when I get home, I will try to find the offending cipher(s), and come up with a repro.

[BJReplay]

OK, @BlackDex I've narrowed it down to two ciphers that cause the existing Android (2024.7.1 11086) and new Beta (2024.8.1 19099) client to fail to sync.

It took a while, but I restored, and deleted ciphers until I could sync, at which point I knew that the deleted ciphers were the bad ciphers preventing syncing.

The new client displays a message under my vault We were unable to process your request. Please try again later or contact us. Try again (where Try again is hyperlink style button).

The two suspect cyphers are (listed using BW CLI bw list items --organizationid null

[{"passwordHistory":[{"lastUsedDate":null,"password":"history item 1"},{"lastUsedDate":null,"password":"history item 2"},{"lastUsedDate":null,"password":"history item 3"},{"lastUsedDate":null,"password":"history item 4 "}],"revisionDate":"2024-09-18T05:28:44.903Z","creationDate":"2023-12-23T01:28:13.392Z","deletedDate":null,"object":"item","id":"item guid redacted","organizationId":null,"folderId":"gilder guid redacted","type":1,"reprompt":0,"name":"oracle.com","notes":null,"favorite":false,"login":{"fido2Credentials":[],"uris":[{"match":null,"uri":"https://signup.cloud.oracle.com"},{"match":null,"uri":"https://identity.oraclecloud.com"}],"username":"email redacted","password":"password redacted","totp":"TOTP KEY Present and Redacted","passwordRevisionDate":null},"collectionIds":[]},{"passwordHistory":[{"lastUsedDate":"2023-12-22T14:34:25.153Z","password":"password history 1"},{"lastUsedDate":"2023-12-22T14:34:14.839Z","password":"password history 2"},{"lastUsedDate":"1970-01-01T00:00:00.000Z","password":"T3mp0@rryP@55w0rd"}],"revisionDate":"2024-09-18T05:28:45.131Z","creationDate":"2023-12-23T01:28:13.391Z","deletedDate":null,"object":"item","id":"ID GUID redacted","organizationId":null,"folderId":"Folder GUID redacted","type":1,"reprompt":0,"name":"oracle.com - Backup","notes":null,"favorite":false,"login":{"fido2Credentials":[],"uris":[{"match":null,"uri":"https://signup.cloud.oracle.com"},{"match":null,"uri":"https://identity.oraclecloud.com"}],"username":"email redacted","password":"password Redacted","totp":"TOPT Present and Redacted","passwordRevisionDate":null},"collectionIds":[]}]

Using the latest testing build Version 1.32.0-6ceed928 still doesn't allow the ciphers to sync.

Deleting the first cipher listed above with the null lastUsedDate in the password history allows the remaining item (with a 1/1/1970 - so near enough to null last used date to sync.

At a guess those history items were imported when I transitioned from cough lastpass with null dates via an import, but interesting nonetheless.

Do you want me to open a new issue or is this edge case enough (Null Dates on Password History) that you're happy to let sleeping dogs lie)?

[BlackDex]

@BJReplay Thanks, i was able to solve this :)

[BJReplay]

@BJReplay Thanks, i was able to solve this :)

Should that fix be in the latest :testing build on docker from 8 hours ago?

The sync error still exists with that build

[BlackDex]

@BJReplay it should indeed. That should at least resolve the null time issues which i was able to replicate.

If it doesn't, then something else is wrong too.

[BJReplay]

@BlackDex OK, I will restore and examine the ciphers again tomorrow - it is possible that there were more nulls in other fields so I am thinking of working out how to review the ciphers via BW CLI look for nulls.

[BlackDex]

@BJReplay Looking at what you posted here, it doesn't look like anything else should be causing issues.
I would suggest to follow the steps i wrote down here: #4870 (comment)

It doesn't matter which client get's the sync, all clients receive the same sync data, so the exact data from the web-vault should be fine too.

[BJReplay]

OK, will do.

I've rotated the passwords for the production accounts, so I know that they're OK, and I'll disable and re-enable the TOTP sign on, so most of the data is fine to share in any case, so I'll try to get the data to you in the next couple of days.

[BJReplay]

@BlackDex I can now repro and will attempt to provide the information as per the steps in #4870 as requested, but can provide information to assist:

1. Retrieving the cipher fails whether or not it is in trash.
2. Restoring from trash to the vault still causes the vault sync to fail on both legacy and native android apps.
3. Saving anything on the cipher fixes the issue - the password history dates are updated to 1/1/1970.

I haven't reviewed your code, but that makes sense: you won't change a cipher on the database - it is what it is, but when you encounter a corrupt one, you fix null / invalid dates with a valid default.

So, when I went to redact the cipher before capturing, and saved it with a redacted email, the null dates in the password history were written back as 1/1/1970, and suddenly my clients could sync.

However, until then, only the web client would successfully read that cipher.

Before I repro and attempt to capture, I'll see if I can see from your commit if the changes were only on saving a cipher, rather than on retrieving a cypher as well, as, from what I can see, the nulls only get replaced on saving.

[BJReplay]

@BlackDex I have a logcat zipped - I will get it to you.

I have rolled the TOTP and passwords in the logcat so the only sensitive information in the capture are my emails.

Can confirm UI and vault export and mobile clients all receive null lastUsedDate for password history until some other attribute of a cipher is changed, at which point all nulls are updated to 1/1/1970.

Logcat is from a few seconds before initial log in to qa build as linked above, to the error message We were unable to process your request. Please try again later or contact us. Try again

adb logcat -T "2024-09-20 12:42:00.000" > vaultwardennullpw.log

vaultwarden  | /--------------------------------------------------------------------\
vaultwarden  | |                        Starting Vaultwarden                        |
vaultwarden  | |                      Version 1.32.0-0bd8f607                       |
vaultwarden  | |--------------------------------------------------------------------|
vaultwarden  | | This is an *unofficial* Bitwarden implementation, DO NOT use the   |
vaultwarden  | | official channels to report bugs/features, regardless of client.   |
vaultwarden  | | Send usage/configuration questions or feature requests to:         |
vaultwarden  | |   https:/dani-garcia/vaultwarden/discussions or        |
vaultwarden  | |   https://vaultwarden.discourse.group/                             |
vaultwarden  | | Report suspected bugs/issues in the software itself at:            |
vaultwarden  | |   https:/dani-garcia/vaultwarden/issues/new            |
vaultwarden  | \--------------------------------------------------------------------/
vaultwarden  |

bw list items --pretty
[
  {
    "passwordHistory": [
      {
        "lastUsedDate": null,
        "password": "u*vg!hrmW2zx3seMvgrHFFR8UUf2t6Ya1"
      },
      {
        "lastUsedDate": null,
        "password": "u*vg!hrmW2zx3seMvgrHFFR8UUf2t6Ya"
      },
      {
        "lastUsedDate": null,
        "password": "IAlHE164ybU*7B*xVC2*2r##axnO9%w0"
      },
      {
        "lastUsedDate": null,
        "password": "!u%0ZLEcp1PCxq%IFbWzlaU^aX@X7U4b"
      }
    ],
    "revisionDate": "2024-09-20T02:23:48.933Z",
    "creationDate": "2023-12-23T01:28:13.392Z",
    "deletedDate": null,
    "object": "item",
    "id": "040bcf7c-bb9e-4ba2-8dc9-3bbfa08984ca",
    "organizationId": null,
    "folderId": "49ba8baa-db24-43cf-bc2a-20c6a055b24b",
    "type": 1,
    "reprompt": 0,
    "name": "oracle.com",
    "notes": null,
    "favorite": false,
    "login": {
      "fido2Credentials": [],
      "uris": [
        {
          "match": null,
          "uri": "https://signup.cloud.oracle.com"
        },
        {
          "match": null,
          "uri": "https://identity.oraclecloud.com"
        }
      ],
      "username": "[email protected]",
      "password": "redacted",
      "totp": "redacted",
      "passwordRevisionDate": null
    },
    "collectionIds": []
  }
]

Vault Export from Web UI:

{
  "encrypted": false,
  "folders": [
    {
      "id": "49ba8baa-db24-43cf-bc2a-20c6a055b24b",
      "name": "Development"
    }
  ],
  "items": [
    {
      "passwordHistory": [
        {
          "lastUsedDate": null,
          "password": "u*vg!hrmW2zx3seMvgrHFFR8UUf2t6Ya1"
        },
        {
          "lastUsedDate": null,
          "password": "u*vg!hrmW2zx3seMvgrHFFR8UUf2t6Ya"
        },
        {
          "lastUsedDate": null,
          "password": "IAlHE164ybU*7B*xVC2*2r##axnO9%w0"
        },
        {
          "lastUsedDate": null,
          "password": "!u%0ZLEcp1PCxq%IFbWzlaU^aX@X7U4b"
        }
      ],
      "revisionDate": "2024-09-20T02:23:48.933Z",
      "creationDate": "2023-12-23T01:28:13.392Z",
      "deletedDate": null,
      "id": "040bcf7c-bb9e-4ba2-8dc9-3bbfa08984ca",
      "organizationId": null,
      "folderId": "49ba8baa-db24-43cf-bc2a-20c6a055b24b",
      "type": 1,
      "reprompt": 0,
      "name": "oracle.com",
      "notes": null,
      "favorite": false,
      "login": {
        "fido2Credentials": [],
        "uris": [
          {
            "match": null,
            "uri": "https://signup.cloud.oracle.com"
          },
          {
            "match": null,
            "uri": "https://identity.oraclecloud.com"
          }
        ],
        "username": "[email protected]",
        "password": "redacted",
        "totp": "redacted"
      },
      "collectionIds": null
    }
  ]
}

[BlackDex]

But, i have fixed that already 🤔.

Very strange, the null dates should not have been returned using that version of Vaultwarden. They should all be epoch 0 time.

You send me the adb logs, so ill check out those and see if i can find anything there.