Fsync state file before exposing it (#30929)

With multiple data paths, we write the state files for index metadata to all data paths. We only properly fsync on the first location, though. For other locations, we possibly expose the file before its contents is properly fsynced. This can lead to situations where, after a crash, and where the first data path is not available anymore, ES will see a partially-written state file, preventing the node to start up.

server/src/main/java/org/elasticsearch/gateway/MetaDataStateFormat.java

@@ -141,9 +141,10 @@ public void close() throws IOException {
  Path finalPath = stateLocation.resolve(fileName);
  try {
  Files.copy(finalStatePath, tmpPath);
+ IOUtils.fsync(tmpPath, false); // fsync the state file
  // we are on the same FileSystem / Partition here we can do an atomic move
  Files.move(tmpPath, finalPath, StandardCopyOption.ATOMIC_MOVE);
- IOUtils.fsync(stateLocation, true); // we just fsync the dir here..
+ IOUtils.fsync(stateLocation, true);
  } finally {
  Files.deleteIfExists(tmpPath);
  }