Currently broken bnfc-system-tests #280

andreasabel · 2019-12-17T16:11:29Z

This is an issue documenting known bugs of BNFC that cause some system tests to fail. We keep the tests in the test suite nevertheless, to stimulate further bug fixing.

Currently known issues for some of the backends:

Use white-space as separator #70: whitespace separators
Support block comment delimiters > 2 characters in Haskell backend #202: long block comment delimiters (Ocaml)
C -family backends generate incorrect parser #235: position tokens not supported by some backends, see Position tokens not supported by cpp-nostl backend #297
support for utf-8 / unicode characters #249: unicode
RegExp: Nested subtractions are translated wrongly to Alex #256: Regex: nested subtractions (C++)
Some backends do not implement the define pragma #266: define pragma
Make backends more resilient against keyword and Prelude clashes #278: keyword clashes
Error location missing with recent menhir (starting with 2021-12-30 release) #414: error handling in Menhir, affects 100_coercion_lists

Test run of 2019-12-16:

Tests: 484
Passed: 436
Pending: 0
Failures: 14
Errors: 34
Timed out: 0
Filtered: 0
Failures:
- Parameterized tests:Java (with antlr):Java (with antlr):70_WhiteSpaceSeparator
- Parameterized tests:Java (with jflex and line numbers):Java (with jflex and line numbers):70_WhiteSpaceSeparator
- Parameterized tests:Java (with jflex):Java (with jflex):70_WhiteSpaceSeparator
- Parameterized tests:Java (with namespace):Java (with namespace):70_WhiteSpaceSeparator
- Parameterized tests:Java (with line numbers):Java (with line numbers):70_WhiteSpaceSeparator
- Parameterized tests:Java:Java:70_WhiteSpaceSeparator
- Parameterized tests:OCaml:OCaml:70_WhiteSpaceSeparator
- Parameterized tests:C++ (no STL):C++ (no STL):249_unicode
- Parameterized tests:C++ (no STL):C++ (no STL):70_WhiteSpaceSeparator
- Parameterized tests:C++ (with namespace):C++ (with namespace):249_unicode
- Parameterized tests:C++ (with namespace):C++ (with namespace):70_WhiteSpaceSeparator
- Parameterized tests:C++:C++:249_unicode
- Parameterized tests:C++:C++:70_WhiteSpaceSeparator
- Parameterized tests:C:C:70_WhiteSpaceSeparator
Errors:
- Parameterized tests:Java (with antlr):Java (with antlr):249_unicode
- Parameterized tests:Java (with antlr):Java (with antlr):278_Keywords
- Parameterized tests:Java (with jflex and line numbers):Java (with jflex and line numbers):249_unicode
- Parameterized tests:Java (with jflex and line numbers):Java (with jflex and line numbers):278_Keywords
- Parameterized tests:Java (with jflex):Java (with jflex):249_unicode
- Parameterized tests:Java (with jflex):Java (with jflex):278_Keywords
- Parameterized tests:Java (with namespace):Java (with namespace):249_unicode
- Parameterized tests:Java (with namespace):Java (with namespace):278_Keywords
- Parameterized tests:Java (with line numbers):Java (with line numbers):249_unicode
- Parameterized tests:Java (with line numbers):Java (with line numbers):278_Keywords
- Parameterized tests:Java:Java:249_unicode
- Parameterized tests:Java:Java:278_Keywords
- Parameterized tests:OCaml:OCaml:278_Keywords
- Parameterized tests:Haskell/GADT:Haskell/GADT:278_Keywords
- Parameterized tests:Haskell & Agda:Haskell & Agda:278_Keywords
- Parameterized tests:C++ (no STL):C++ (no STL):256_Regex
- Parameterized tests:C++ (no STL):C++ (no STL):278_Keywords
- Parameterized tests:C++ (with namespace):C++ (with namespace):278_Keywords
- Parameterized tests:C++:C++:278_Keywords
- Parameterized tests:C:C:249_unicode
- Parameterized tests:C:C:278_Keywords
- Current parameterized test:Java (with antlr):278_Keywords
- Current parameterized test:Java (with jflex and line numbers):278_Keywords
- Current parameterized test:Java (with jflex):278_Keywords
- Current parameterized test:Java (with namespace):278_Keywords
- Current parameterized test:Java (with line numbers):278_Keywords
- Current parameterized test:Java:278_Keywords
- Current parameterized test:OCaml:278_Keywords
- Current parameterized test:Haskell/GADT:278_Keywords
- Current parameterized test:Haskell & Agda:278_Keywords
- Current parameterized test:C++ (no STL):278_Keywords
- Current parameterized test:C++ (with namespace):278_Keywords
- Current parameterized test:C++:278_Keywords
- Current parameterized test:C:278_Keywords

Total execution time: 3815198ms

The text was updated successfully, but these errors were encountered:

andreasabel · 2020-01-02T20:27:46Z

2020-01-02

Tests: 512
Passed: 484
Pending: 0
Failures: 15
Errors: 13
Timed out: 0
Filtered: 0
Failures:
- Parameterized tests:Java (with antlr):Java (with antlr):70_WhiteSpaceSeparator
- Parameterized tests:Java (with jflex and line numbers):Java (with jflex and line numbers):70_WhiteSpaceSeparator
- Parameterized tests:Java (with jflex):Java (with jflex):70_WhiteSpaceSeparator
- Parameterized tests:Java (with namespace):Java (with namespace):70_WhiteSpaceSeparator
- Parameterized tests:Java (with line numbers):Java (with line numbers):70_WhiteSpaceSeparator
- Parameterized tests:Java:Java:70_WhiteSpaceSeparator
- Parameterized tests:OCaml/Menhir:OCaml/Menhir:70_WhiteSpaceSeparator
- Parameterized tests:OCaml:OCaml:70_WhiteSpaceSeparator
- Parameterized tests:C++ (with namespace):C++ (with namespace):249_unicode
- Parameterized tests:C++ (with namespace):C++ (with namespace):70_WhiteSpaceSeparator
- Parameterized tests:C:C:70_WhiteSpaceSeparator
- Parameterized tests:C++:C++:249_unicode
- Parameterized tests:C++:C++:70_WhiteSpaceSeparator
- Parameterized tests:C++ (no STL):C++ (no STL):249_unicode
- Parameterized tests:C++ (no STL):C++ (no STL):70_WhiteSpaceSeparator
Errors:
- Layout parsing test:Haskell & Agda:194_layout
- Parameterized tests:Java (with antlr):Java (with antlr):249_unicode
- Parameterized tests:Java (with jflex and line numbers):Java (with jflex and line numbers):249_unicode
- Parameterized tests:Java (with jflex):Java (with jflex):249_unicode
- Parameterized tests:Java (with namespace):Java (with namespace):249_unicode
- Parameterized tests:Java (with line numbers):Java (with line numbers):249_unicode
- Parameterized tests:Java:Java:249_unicode
- Parameterized tests:OCaml/Menhir:OCaml/Menhir:202_comments
- Parameterized tests:OCaml/Menhir:OCaml/Menhir:278_Keywords
- Parameterized tests:OCaml:OCaml:202_comments
- Parameterized tests:OCaml:OCaml:278_Keywords
- Parameterized tests:C:C:249_unicode
- Parameterized tests:C++ (no STL):C++ (no STL):256_Regex

Total execution time: 3688716ms

andreasabel · 2020-01-03T12:22:07Z

2020-01-03

Tests: 529
Passed: 490
Pending: 0
Failures: 19
Errors: 20
Timed out: 0
Filtered: 0
Failures:
- Parameterized tests:Java (with antlr):Java (with antlr):70_WhiteSpaceSeparator
- Parameterized tests:Java (with jflex and line numbers):Java (with jflex and line numbers):70_WhiteSpaceSeparator
- Parameterized tests:Java (with jflex):Java (with jflex):70_WhiteSpaceSeparator
- Parameterized tests:Java (with namespace):Java (with namespace):70_WhiteSpaceSeparator
- Parameterized tests:Java (with line numbers):Java (with line numbers):70_WhiteSpaceSeparator
- Parameterized tests:Java:Java:70_WhiteSpaceSeparator
- Parameterized tests:OCaml/Menhir:OCaml/Menhir:70_WhiteSpaceSeparator
- Parameterized tests:OCaml:OCaml:70_WhiteSpaceSeparator
- Parameterized tests:C:C:70_WhiteSpaceSeparator
- Parameterized tests:C++:C++:249_unicode
- Parameterized tests:C++:C++:70_WhiteSpaceSeparator
- Parameterized tests:C++:C++:266_define
- Parameterized tests:C++ (no STL):C++ (no STL):249_unicode
- Parameterized tests:C++ (no STL):C++ (no STL):70_WhiteSpaceSeparator
- Parameterized tests:C++ (with namespace):C++ (with namespace):249_unicode
- Parameterized tests:C++ (with namespace):C++ (with namespace):70_WhiteSpaceSeparator
- Parameterized tests:C++ (with namespace):C++ (with namespace):266_define
- Current parameterized test:C++:266_define
- Current parameterized test:C++ (with namespace):266_define
Errors:
- Parameterized tests:Java (with antlr):Java (with antlr):249_unicode
- Parameterized tests:Java (with antlr):Java (with antlr):266_define
- Parameterized tests:Java (with jflex and line numbers):Java (with jflex and line numbers):249_unicode
- Parameterized tests:Java (with jflex):Java (with jflex):249_unicode
- Parameterized tests:Java (with namespace):Java (with namespace):249_unicode
- Parameterized tests:Java (with line numbers):Java (with line numbers):249_unicode
- Parameterized tests:Java:Java:249_unicode
- Parameterized tests:OCaml/Menhir:OCaml/Menhir:202_comments
- Parameterized tests:OCaml/Menhir:OCaml/Menhir:278_Keywords
- Parameterized tests:OCaml:OCaml:202_comments
- Parameterized tests:OCaml:OCaml:278_Keywords
- Parameterized tests:C:C:249_unicode
- Parameterized tests:C:C:266_define
- Parameterized tests:Haskell (with functor):Haskell (with functor):266_define
- Parameterized tests:C++ (no STL):C++ (no STL):256_Regex
- Parameterized tests:C++ (no STL):C++ (no STL):266_define
- Current parameterized test:Java (with antlr):266_define
- Current parameterized test:C:266_define
- Current parameterized test:Haskell (with functor):266_define
- Current parameterized test:C++ (no STL):266_define

Total execution time: 4015888ms

andreasabel · 2020-10-05T10:24:03Z

2020-10-05

Tests: 537
Passed: 523
Pending: 0
Failures: 0
Errors: 14
Timed out: 0
Filtered: 0
Errors:
- Pygments backend:Pygments Backend
- Parameterized tests:Java (with jflex and line numbers):Java (with jflex and line numbers):235_SymbolsOverlapTokens
- Parameterized tests:Java (with jflex):Java (with jflex):235_SymbolsOverlapTokens
- Parameterized tests:OCaml/Menhir:OCaml/Menhir:289_LexerKeywords
- Parameterized tests:OCaml/Menhir:OCaml/Menhir:202_comments
- Parameterized tests:OCaml/Menhir:OCaml/Menhir:278_Keywords
- Parameterized tests:C++ (with namespace):C++ (with namespace):235_SymbolsOverlapTokens
- Parameterized tests:C++ (no STL):C++ (no STL):235_SymbolsOverlapTokens
- Parameterized tests:OCaml:OCaml:202_comments
- Parameterized tests:OCaml:OCaml:278_Keywords
- Current parameterized test:Java (with jflex and line numbers):235_SymbolsOverlapTokens
- Current parameterized test:Java (with jflex):235_SymbolsOverlapTokens
- Current parameterized test:C++ (with namespace):235_SymbolsOverlapTokens
- Current parameterized test:C++ (no STL):235_SymbolsOverlapTokens

Total execution time: 4627800ms

andreasabel · 2020-10-06T21:49:39Z

2020-10-06

Tests: 555
Passed: 542
Pending: 0
Failures: 7
Errors: 6
Timed out: 0
Filtered: 0
Failures:
- Testing BNFC on ill-formed LBNF input:fail-lbnf/254-empty-input
- Testing BNFC on ill-formed LBNF input:fail-lbnf/214-superfluous-character
- Testing BNFC on ill-formed LBNF input:fail-lbnf/214-undefined-defined-label
- Testing BNFC on ill-formed LBNF input:fail-lbnf/214-define-only-label
- Testing BNFC on ill-formed LBNF input:fail-lbnf/214-define-uppercase
- Testing BNFC on ill-formed LBNF input:fail-lbnf/214-define-only-label
- Testing BNFC on well-formed LBNF input:succeed-lbnf/duplicate-label-same-cat
Errors:
- Pygments backend:Pygments Backend
- Haskell/CNF:Examples:java
- Parameterized tests:Java (with antlr):Examples:java
- Parameterized tests:OCaml/Menhir:OCaml/Menhir:202_comments
- Parameterized tests:C:Examples:java
- Parameterized tests:OCaml:OCaml:202_comments

Total execution time: 4161630ms

andreasabel · 2020-10-07T13:05:15Z

2020-10-07

Tests: 555
Passed: 549
Pending: 0
Failures: 1
Errors: 5
Timed out: 0
Filtered: 0
Failures:
- Regression tests:Shouldn't print all categories in error message #151 Shouldn't print all categories in error message
Errors:
- Pygments backend:Pygments Backend
- Haskell/CNF:Examples:java
- Parameterized tests:Java (with antlr):Examples:java
- Parameterized tests:OCaml/Menhir:OCaml/Menhir:202_comments
- Parameterized tests:OCaml:OCaml:202_comments

Total execution time: 7037705ms (had a zoom meeting running)

andreasabel · 2020-10-08T11:07:57Z

2020-10-08

Tests: 549
Passed: 547
Errors:
- Parameterized tests:OCaml/Menhir:OCaml/Menhir:202_comments
- Parameterized tests:OCaml:OCaml:202_comments

Again:

Tests: 567
Passed: 565
Errors:
- Parameterized tests:OCaml/Menhir:OCaml/Menhir:202_comments
- Parameterized tests:OCaml:OCaml:202_comments

Total execution time: 4253609ms

2020-12-31: still the same

andreasabel · 2021-05-02T17:39:57Z

New breakage is Parameterized tests:Haskell/GADT:Haskell/GADT:289_LexerKeywords because categories ERROR and ERROR_ introduce a clash with their generated tags ERROR_ and ERROR__.

andreasabel · 2021-05-18T13:55:11Z

The test for examples/define/test.cf is broken for the C backend, because of #363 (comment).

andreasabel · 2022-02-10T17:27:38Z

2022-02-10:

Tests: 627
Passed: 620
Failures: 2
Errors: 5
Failures:
- Parameterized tests:OCaml/Menhir:OCaml/Menhir:100_coercion_lists
- Testing BNFC on ill-formed LBNF input:fail-lbnf/duplicate-cons
Errors:
- Parameterized tests:Haskell/GADT:Haskell/GADT:289_LexerKeywords
- Parameterized tests:OCaml:OCaml:289_LexerKeywords
- Parameterized tests:OCaml:OCaml:202_comments
- Parameterized tests:OCaml/Menhir:OCaml/Menhir:289_LexerKeywords
- Parameterized tests:OCaml/Menhir:OCaml/Menhir:202_comments

Total execution time: 5354297ms

andreasabel · 2022-12-14T15:59:09Z

2022-12-14:

Tests: 646
Passed: 640
Failures: 1
Errors: 5
Failures:
- Testing BNFC on ill-formed LBNF input:fail-lbnf/duplicate-cons
Errors:
- Parameterized tests:Haskell/GADT:Haskell/GADT:289_LexerKeywords
- Parameterized tests:OCaml:OCaml:289_LexerKeywords
- Parameterized tests:OCaml:OCaml:202_comments
- Parameterized tests:OCaml/Menhir:OCaml/Menhir:289_LexerKeywords
- Parameterized tests:OCaml/Menhir:OCaml/Menhir:202_comments

Total execution time: 4805463ms

andreasabel · 2023-07-26T14:42:41Z

2023-07-26:

Tests: 627
Passed: 620
Pending: 0
Failures: 1
Errors: 6
Timed out: 0
Filtered: 0
Failures:
- Testing BNFC on ill-formed LBNF input:fail-lbnf/duplicate-cons
Errors:
- Parameterized tests:Haskell/GADT:Haskell/GADT:289_LexerKeywords
- Parameterized tests:Java (with antlr):Examples:C
- Parameterized tests:OCaml:OCaml:289_LexerKeywords
- Parameterized tests:OCaml:OCaml:202_comments
- Parameterized tests:OCaml/Menhir:OCaml/Menhir:289_LexerKeywords
- Parameterized tests:OCaml/Menhir:OCaml/Menhir:202_comments

andreasabel · 2023-07-26T19:50:23Z

2023-07-26 after fixing #455 and #451:

Tests: 646
Passed: 640
Pending: 0
Failures: 1
Errors: 5
Timed out: 0
Filtered: 0
Failures:
- Testing BNFC on ill-formed LBNF input:fail-lbnf/duplicate-cons
Errors:
- Parameterized tests:Haskell/GADT:Haskell/GADT:289_LexerKeywords
- Parameterized tests:OCaml:OCaml:289_LexerKeywords
- Parameterized tests:OCaml:OCaml:202_comments
- Parameterized tests:OCaml/Menhir:OCaml/Menhir:289_LexerKeywords
- Parameterized tests:OCaml/Menhir:OCaml/Menhir:202_comments

andreasabel added the Testing label Dec 17, 2019

andreasabel mentioned this issue Oct 8, 2020

Support block comment delimiters > 2 characters in Haskell backend #202

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Currently broken bnfc-system-tests #280

Currently broken bnfc-system-tests #280

andreasabel commented Dec 17, 2019 •

edited

Loading

andreasabel commented Jan 2, 2020

andreasabel commented Jan 3, 2020

andreasabel commented Oct 5, 2020

andreasabel commented Oct 6, 2020

andreasabel commented Oct 7, 2020

andreasabel commented Oct 8, 2020 •

edited

Loading

andreasabel commented May 2, 2021

andreasabel commented May 18, 2021

andreasabel commented Feb 10, 2022

andreasabel commented Dec 14, 2022

andreasabel commented Jul 26, 2023

andreasabel commented Jul 26, 2023

Currently broken bnfc-system-tests #280

Currently broken bnfc-system-tests #280

Comments

andreasabel commented Dec 17, 2019 • edited Loading

andreasabel commented Jan 2, 2020

andreasabel commented Jan 3, 2020

andreasabel commented Oct 5, 2020

andreasabel commented Oct 6, 2020

andreasabel commented Oct 7, 2020

andreasabel commented Oct 8, 2020 • edited Loading

andreasabel commented May 2, 2021

andreasabel commented May 18, 2021

andreasabel commented Feb 10, 2022

andreasabel commented Dec 14, 2022

andreasabel commented Jul 26, 2023

andreasabel commented Jul 26, 2023

andreasabel commented Dec 17, 2019 •

edited

Loading

andreasabel commented Oct 8, 2020 •

edited

Loading