Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

double free or corruption (!prev) error on Json push_back and write #2632

Closed
2 of 3 tasks
nhanders opened this issue Feb 11, 2021 · 10 comments
Closed
2 of 3 tasks

double free or corruption (!prev) error on Json push_back and write #2632

nhanders opened this issue Feb 11, 2021 · 10 comments
Labels
kind: bug solution: invalid the issue is not related to the library

Comments

@nhanders
Copy link

nhanders commented Feb 11, 2021

What is the issue you have?

When adding nlohmann::json (Json) elements to a variable using push_back, we occasionally get the error double free or corruption (!prev). The error free(): invalid next size (normal) also occurred when trying to write the Json variable to file using std::ofstream.

The Json files we are writing are roughly half a MB in size.

We are passing the nlohmann::json variables by value to push_back, so threading should not be an issue.

The stack traces of two of these crashes can be found here and here

Please describe the steps to reproduce the issue.

We have not been able to consistently reproduce the error.

Can you provide a small but working code example?

using Json = nlohmann::json;

const Json GetGeoJson() {
  Json large_position_geojson = {
    { "foo", "bar" }
  };
  return large_position_geojson;
}

int main() {
  std::vector<Json> map_geojson;
  map_geojson.push_back(GetGeoJson());

  std::ofstream json_file("file.geojson");
  json_file << std::setw(2) << map_geojson << std::endl;
}

Which compiler and operating system are you using?

  • Compiler: gcc-9, g++-9
  • Operating system: Ubuntu 18.04.5 LTS

Which version of the library did you use?

  • latest release version 3.9.1
  • other release - please state the version: ___
  • the develop branch, commit hash f4155e47278100296ba2a8d149cf4ae5b3581e44
@nlohmann
Copy link
Owner

I do not believe this is an issue in the library as it was and is tested extensively using address sanitizer and Valgrind. Can you share the file file.geojson?

@nhanders
Copy link
Author

Sure, here is the file as a .txt file (GitHub doesn't allow uploading .geojson files).
file.txt

@nickaein
Copy link
Contributor

nickaein commented Feb 22, 2021

I couldn't reproduce the bug with the sample code. I've also tested it using AddressSanitizer and it didn't find any memory corruption.

Can you try Sanitizers/Valgrind to track down the issue? They can be very helpful is such cases.

This was my setup:
Library version: 3.9.1
Compiler version: 9.3.0-17ubuntu1
Compiler command: g++ -fsanitize=address -fno-omit-frame-pointer a.cpp
The (modified) sample code that I ran:

#include "json.hpp"
#include <fstream>
#include <vector>
#include <iomanip>

using Json = nlohmann::json;

const Json GetGeoJson()
{
    std::ifstream fin("FruitScan.txt");
    Json large_position_geojson;
    fin >> large_position_geojson;

    return large_position_geojson;
}

int main()
{
    std::vector<Json> map_geojson;
    map_geojson.push_back(GetGeoJson());

    std::ofstream json_file("file.geojson");
    json_file << std::setw(2) << map_geojson << std::endl;
}

P.S: These random crashes can be caused by a bug somewhere in the program that is corrupting the JSON object.

Also, do you have enough memory available? An ~500KB raw input can translate to a JSON object of size in dozen of MBs which can be problematic on some memory-constrained platforms.

@nhanders
Copy link
Author

@nickaein your point on memory availability is very interesting, and may be the cause of our issue. As you pointed out, we are running on a "memory-constrained platform".

Is there a tool that we could use to confirm this hypothesis? Maybe AddressSanitizer/Valgrind?

I'm now pretty confident that this issue is on our side - possibly a memory availability issue - so I will close. Thanks for all the helpful feedback!

@nlohmann nlohmann added the solution: invalid the issue is not related to the library label Mar 23, 2021
@nlwmode
Copy link

nlwmode commented Sep 15, 2022

I had a similar problem:
free(): double free detected in tcache 2
Aborted (core dumped)

Note:
It is all fine when I compile my code by g++-7 under release and debug-mode.
But it will cause the following problem when I use g++-9 or g++11 under release-mode; but it is OK under debug-mode even if I use AddressSanitizer/Valgrind for memory check.

OS environment for error condition:
Compiler: gcc-11, g++-11
Operating system: Ubuntu 18.04.5 LTS
Release mode compiler option: -Wall -Wno-unknown-pragmas -O3

My code is below:

#include "json/nlohmann/json.hpp"
#include <assert.h>

class configer{
public:
 
  /**
   * @brief mainly a parser for read the configuration
   * 
   * @param file 
   */
  explicit configer(std::string file) 
    : _config_file(file)
  {
    std::ifstream ifs(_config_file);
    _data = nlohmann::json::parse(ifs);
  }

public:
  /**
   * @brief Get the value object
   * 
   * @tparam T 
   * @param param_level the params follow the hierarchical relationship
   * 
   * 
   * @return T 
   */
  template<typename T>
  T get_value( std::vector<std::string> param_level){
    nlohmann::json value = _data;
    for(uint i = 0; i < param_level.size(); ++i){
      value = value[ param_level[i] ];
    }
    if( !value.is_null() ){
      return value.get<T>();
    }
    else{
      std::cerr << "config file is wrong!" << std::endl;
      assert(false);
      return (T)NULL;
    }
  }

private:
  nlohmann::json   _data;
  std::string      _config_file;
  
};  // end class configer

@nlohmann
Copy link
Owner

@nlwmode There is no tcache function in the library. Can you share more information?

@falbrechtskirchinger
Copy link
Contributor

@nlohmann That's just a glibc error. (tcache is an optimization in the glibc heap allocator.)

@nlwmode
Copy link

nlwmode commented Sep 15, 2022

@falbrechtskirchinger @nlohmann There is almost all the information of my environment, code and reports. Sure, it is a glibc error. And I also searched that glibc-2.27 allowed tache2 for "double free", but the glibc(2.23, 2.29,-above) do not allowed. We can solve this problem by using glibc-2.27 to compile codes because it allows "double free". We can using command "ldd --version" to check our glibc version.
But maybe there is still a problem for current json of the other version of glibc ?!.

@falbrechtskirchinger
Copy link
Contributor

falbrechtskirchinger commented Sep 15, 2022

This library does minimal manual memory management with code that hasn't changed for a while and is tested with sanitizers and Valgrind.

I've yet to see a memory error that's not the user's fault. If you have a self-contained, minimal reproduction example for the error, please share it.

@nlwmode
Copy link

nlwmode commented Oct 26, 2022

This library does minimal manual memory management with code that hasn't changed for a while and is tested with sanitizers and Valgrind.

I've yet to see a memory error that's not the user's fault. If you have a self-contained, minimal reproduction example for the error, please share it.

It is my fault that I write a function "bool configer()" but do not give a return value.
It is ok when I compile my code by g++7, but g++-9/10/11 give a double free or segmental fault error.
@nhanders So I guess it maybe that your "int main()" function do not give a return value. And you can add the compiler option "-Wall -Werror=return-type" to detected this condition.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind: bug solution: invalid the issue is not related to the library
Projects
None yet
Development

No branches or pull requests

5 participants