Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

perf: Improve output speed by caching author_list #1090

Closed
wants to merge 2 commits into from
Closed

perf: Improve output speed by caching author_list #1090

wants to merge 2 commits into from

Conversation

fukusuket
Copy link
Collaborator

@fukusuket fukusuket commented Jun 8, 2023 •
edited
Loading

What Changed

  • Related Speed up results summary calculation #1088
  • Improve output speed by caching author_list when output
    • Once the author name is read from the file, it will be read from the cache from the second time onwards.
    • The result is less file IO and more speed.

This fix may not improve memory usage. In that case, it will be handled as a separate issue.
I would appreciate it if you could review🙏

Evidence

Environment

  • OS: macOS montery version 13.1
  • Hard: Macbook Air(M1, 2020) , Memory 8GB, Core 8

Test(all-evtx.tgz(6.1GB))

I confirmed that there is no difference result file and output speed improved.

main

fukusuke@fukusukenoMacBook-Air hayabusa-2.5.1-all-platforms % ./hayabusa csv-timeline -d ../all-evtx -o bug.csv --debug

╔╗ ╔╦═══╦╗  ╔╦═══╦══╗╔╗ ╔╦═══╦═══╗
║║ ║║╔═╗║╚╗╔╝║╔═╗║╔╗║║║ ║║╔═╗║╔═╗║
║╚═╝║║ ║╠╗╚╝╔╣║ ║║╚╝╚╣║ ║║╚══╣║ ║║
║╔═╗║╚═╝║╚╗╔╝║╚═╝║╔═╗║║ ║╠══╗║╚═╝║
║║ ║║╔═╗║ ║║ ║╔═╗║╚═╝║╚═╝║╚═╝║╔═╗║
╚╝ ╚╩╝ ╚╝ ╚╝ ╚╝ ╚╩═══╩═══╩═══╩╝ ╚╝
   by Yamato Security

Start time: 2023/06/08 00:05

Total event log files: 1858
Total file size: 6.1 GB

Loading detections rules. Please wait.

Excluded rules: 30
Noisy rules: 7 (Disabled)

Deprecated rules: 166 (4.51%) (Disabled)
Experimental rules: 1965 (53.37%)
Stable rules: 225 (6.11%)
Test rules: 1488 (40.41%)
Undefined rules: 4 (0.11%)
Unsupported rules: 43 (1.17%) (Disabled)

Hayabusa rules: 152
Sigma rules: 3530
Total enabled detection rules: 3682

Output profile: standard

Scanning in progress. Please wait.

1858 / 1858 [=======================================================================================================================] 100.00 %

Scanning finished. Please wait while the results are being saved.

Rule Authors:

╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Zach Mathis (71)                  frack113 (29)              Florian Roth (17)                 Nasreddine Bencherchali (15)    │
│ oscd.community (12)               Roberto Rodriguez (6)      Roberto Rodriguez @Cyb3r... (6)   OTR (5)                         │
│ Tim Shelton (3)                   Thomas Patzke (3)          Timur Zinniatullin (2)            SOC Prime (2)                   │
│ Daniil Yugoslavskiy (2)           Teymur Kheirkhabarov (2)   Alexandr Yampolskyi (2)           @gott_cyber (1)                 │
│ Connor Martin (1)                 Jonhnathan Ribeiro (1)     James Dickenson (1)               Ecco (1)                        │
│ James Pemberton @4A616D6573 (1)   Sherif Eldeeb (1)          Cybex (1)                         Endgame (1)                     │
│ Open Threat Research (1)          Yusuke Matsui (1)          SCYTHE @scythe_io (1)             D3F7A5105 (1)                   │
│ Timur Zinniatullin oscd.... (1)   @0xrawsec (1)              Dimitrios Slamaris (1)            Gleb Sukhodolskiy (1)           │
│ Jakob Weinzettl (1)               Ilyas Ochkov (1)           Aleksey Potapov (1)               Michael Haag (1)                │
│ JHasenbusch (1)                   Andreas Hunkeler (1)       Sander Wiebing (1)                FPT.EagleEye (1)                │
│ Dmitry Uchakin (1)                Mark Russinovich (1)       Tim Rauch (1)                     Christopher Peacock @sec... (1) │
│ Oddvar Moe (1)                    Mark Woan (1)              James Pemberton@4A616D65... (1)   Matthew Green @mgreen27 (1)     │
│ Bhabesh Raj (1)                   @neu5ron (1)                                                                                 │
╰─────────────────────────────────╌──────────────────────────╌─────────────────────────────────╌─────────────────────────────────╯

Results Summary:
First Timestamp: 2009-07-14 00:56:45.074 -04:00
Last Timestamp: 2022-09-18 10:37:13.088 -04:00

Events with hits / Total events: 1,594,166 / 4,817,181 (Data reduction: 3,223,015 events (66.91%))

Total | Unique detections: 1,627,328 | 156
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 11,634 (0.71%) | 17 (10.90%)
Total | Unique medium detections: 11,016 (0.68%) | 43 (27.56%)
Total | Unique low detections: 1,054,591 (64.81%) | 46 (29.49%)
Total | Unique informational detections: 550,087 (33.80%) | 50 (32.05%)

Dates with most total detections:
critical: n/a, high: 2022-09-18 (3,425), medium: 2022-02-08 (4,670), low: 2022-09-18 (911,357), informational: 2022-03-01 (205,934)

Top 5 computers with most unique detections:
critical: n/a
high: evtx-PC (3), evtx-PC (3), DESKTOP-A8CALR3 (2), DESKTOP-A8CALR3 (2), DESKTOP-6D0DBMB (2)
medium: Agamemnon (7), WIN-TKC15D7KHUR (7), DESKTOP-A8CALR3 (6), DESKTOP-6D0DBMB (6), Agamemnon (6)
low: DESKTOP-A8CALR3 (9), DESKTOP-6D0DBMB (9), DESKTOP-6D0DBMB (8), Agamemnon (7), evtx-PC (6)
informational: DESKTOP-6D0DBMB (31), DESKTOP-A8CALR3 (30), WIN-TKC15D7KHUR (30), WIN-FPV0DSIC9O6.sigma.fr (26), Agamemnon (25)

╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Top critical alerts:                        Top high alerts:                                                    │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                                         File Creation Date Changed to Another Year (10,490)                 │
│ n/a                                         Windows Shell/Scripting Application File Write to Suspicio... (991) │
│ n/a                                         Proc Exec (Non-Exe Filetype) (45)                                   │
│ n/a                                         SysmonEnte Usage (33)                                               │
│ n/a                                         Disabling Windows Event Auditing (29)                               │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts:                          Top low alerts:                                                     │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Raw Access Read (8,544)                     Proc Access (1,020,252)                                             │
│ EVTX Created In Uncommon Location (986)     Possible Timestomping (31,784)                                      │
│ Proc Injection (673)                        System Drawing DLL Load (1,030)                                     │
│ Process Ran With High Privilege (191)       Creation of an Executable by an Executable (382)                    │
│ Use Short Name Path in Command Line (133)   Firewall Rule Modified In The Windows Firewall Exception L... (254) │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                                                                                       │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ File Created (422,037)                      Pipe Created (9,044)                                                │
│ File Deleted (53,696)                       Net Conn (8,755)                                                    │
│ Proc Exec (18,780)                          DNS Query (5,108)                                                   │
│ Pipe Conn (17,062)                          WMI Provider Started (681)                                          │
│ Proc Terminated (12,388)                    WMI Modules Loaded (342)                                            │
╰───────────────────────────────────────────╌─────────────────────────────────────────────────────────────────────╯

Saved file: bug.csv (601.2 MB)

Elapsed time: 00:07:51.136
Rule Parse Processing Time: 00:00:01.155
Analysis Processing Time: 00:06:37.199
Output Processing Time: 00:01:12.780

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     4.0 GiB     4.0 GiB     0           4.0 GiB
 committed:     1.0 GiB     4.0 GiB   201.8 GiB  -197.8 GiB                          ok
     reset:     0
    purged:    22.7 GiB
   touched:    64.2 KiB     8.6 MiB    53.0 GiB   -53.0 GiB                          ok
  segments:    16         138         130           8                                not all freed!
-abandoned:     0           0           0           0                                ok
   -cached:     0           0           0           0                                ok
     pages:     0           0         729.1 Ki   -729.1 Ki                           ok
-abandoned:     0           0           0           0                                ok
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:     0
    resets:     0
    purges:    13.1 Ki
   threads:    16          16           0          16                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:   471.148 s
   process: user: 2213.333 s, system: 40.050 s, faults: 25, rss: 3.3 GiB, commit: 1.0 GiB

This PR

fukusuke@fukusukenoMacBook-Air hayabusa-2.5.1-all-platforms % ./hayabusa csv-timeline -d ../all-evtx -o new.csv --debug

╔╗ ╔╦═══╦╗  ╔╦═══╦══╗╔╗ ╔╦═══╦═══╗
║║ ║║╔═╗║╚╗╔╝║╔═╗║╔╗║║║ ║║╔═╗║╔═╗║
║╚═╝║║ ║╠╗╚╝╔╣║ ║║╚╝╚╣║ ║║╚══╣║ ║║
║╔═╗║╚═╝║╚╗╔╝║╚═╝║╔═╗║║ ║╠══╗║╚═╝║
║║ ║║╔═╗║ ║║ ║╔═╗║╚═╝║╚═╝║╚═╝║╔═╗║
╚╝ ╚╩╝ ╚╝ ╚╝ ╚╝ ╚╩═══╩═══╩═══╩╝ ╚╝
   by Yamato Security

Start time: 2023/06/07 23:44

Total event log files: 1858
Total file size: 6.1 GB

Loading detections rules. Please wait.

Excluded rules: 30
Noisy rules: 7 (Disabled)

Deprecated rules: 166 (4.51%) (Disabled)
Experimental rules: 1965 (53.37%)
Stable rules: 225 (6.11%)
Test rules: 1488 (40.41%)
Undefined rules: 4 (0.11%)
Unsupported rules: 43 (1.17%) (Disabled)

Hayabusa rules: 152
Sigma rules: 3530
Total enabled detection rules: 3682

Output profile: standard

Scanning in progress. Please wait.

1858 / 1858 [=======================================================================================================================] 100.00 %

Scanning finished. Please wait while the results are being saved.

Rule Authors:

╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Zach Mathis (71)                  frack113 (29)              Florian Roth (17)                 Nasreddine Bencherchali (15)    │
│ oscd.community (12)               Roberto Rodriguez (6)      Roberto Rodriguez @Cyb3r... (6)   OTR (5)                         │
│ Tim Shelton (3)                   Thomas Patzke (3)          Timur Zinniatullin (2)            SOC Prime (2)                   │
│ Daniil Yugoslavskiy (2)           Teymur Kheirkhabarov (2)   Alexandr Yampolskyi (2)           @gott_cyber (1)                 │
│ Connor Martin (1)                 Jonhnathan Ribeiro (1)     James Dickenson (1)               Ecco (1)                        │
│ James Pemberton @4A616D6573 (1)   Sherif Eldeeb (1)          Cybex (1)                         Endgame (1)                     │
│ Open Threat Research (1)          Yusuke Matsui (1)          SCYTHE @scythe_io (1)             D3F7A5105 (1)                   │
│ Timur Zinniatullin oscd.... (1)   @0xrawsec (1)              Dimitrios Slamaris (1)            Gleb Sukhodolskiy (1)           │
│ Jakob Weinzettl (1)               Ilyas Ochkov (1)           Aleksey Potapov (1)               Michael Haag (1)                │
│ JHasenbusch (1)                   Andreas Hunkeler (1)       Sander Wiebing (1)                FPT.EagleEye (1)                │
│ Dmitry Uchakin (1)                Mark Russinovich (1)       Tim Rauch (1)                     Christopher Peacock @sec... (1) │
│ Oddvar Moe (1)                    Mark Woan (1)              James Pemberton@4A616D65... (1)   Matthew Green @mgreen27 (1)     │
│ Bhabesh Raj (1)                   @neu5ron (1)                                                                                 │
╰─────────────────────────────────╌──────────────────────────╌─────────────────────────────────╌─────────────────────────────────╯

Results Summary:
First Timestamp: 2009-07-14 00:56:45.074 -04:00
Last Timestamp: 2022-09-18 10:37:13.088 -04:00

Events with hits / Total events: 1,594,166 / 4,817,181 (Data reduction: 3,223,015 events (66.91%))

Total | Unique detections: 1,627,328 | 156
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 11,634 (0.71%) | 17 (10.90%)
Total | Unique medium detections: 11,016 (0.68%) | 43 (27.56%)
Total | Unique low detections: 1,054,591 (64.81%) | 46 (29.49%)
Total | Unique informational detections: 550,087 (33.80%) | 50 (32.05%)

Dates with most total detections:
critical: n/a, high: 2022-09-18 (3,425), medium: 2022-02-08 (4,670), low: 2022-09-18 (911,357), informational: 2022-03-01 (205,934)

Top 5 computers with most unique detections:
critical: n/a
high: evtx-PC (3), evtx-PC (3), DESKTOP-A8CALR3 (2), DESKTOP-A8CALR3 (2), DESKTOP-6D0DBMB (2)
medium: Agamemnon (7), WIN-TKC15D7KHUR (7), DESKTOP-A8CALR3 (6), DESKTOP-6D0DBMB (6), Agamemnon (6)
low: DESKTOP-A8CALR3 (9), DESKTOP-6D0DBMB (9), DESKTOP-6D0DBMB (8), Agamemnon (7), evtx-PC (6)
informational: DESKTOP-6D0DBMB (31), DESKTOP-A8CALR3 (30), WIN-TKC15D7KHUR (30), WIN-FPV0DSIC9O6.sigma.fr (26), Agamemnon (25)

╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Top critical alerts:                        Top high alerts:                                                    │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                                         File Creation Date Changed to Another Year (10,490)                 │
│ n/a                                         Windows Shell/Scripting Application File Write to Suspicio... (991) │
│ n/a                                         Proc Exec (Non-Exe Filetype) (45)                                   │
│ n/a                                         SysmonEnte Usage (33)                                               │
│ n/a                                         Disabling Windows Event Auditing (29)                               │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts:                          Top low alerts:                                                     │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Raw Access Read (8,544)                     Proc Access (1,020,252)                                             │
│ EVTX Created In Uncommon Location (986)     Possible Timestomping (31,784)                                      │
│ Proc Injection (673)                        System Drawing DLL Load (1,030)                                     │
│ Process Ran With High Privilege (191)       Creation of an Executable by an Executable (382)                    │
│ Use Short Name Path in Command Line (133)   Firewall Rule Modified In The Windows Firewall Exception L... (254) │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                                                                                       │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ File Created (422,037)                      Pipe Created (9,044)                                                │
│ File Deleted (53,696)                       Net Conn (8,755)                                                    │
│ Proc Exec (18,780)                          DNS Query (5,108)                                                   │
│ Pipe Conn (17,062)                          WMI Provider Started (681)                                          │
│ Proc Terminated (12,388)                    WMI Modules Loaded (342)                                            │
╰───────────────────────────────────────────╌─────────────────────────────────────────────────────────────────────╯

Saved file: new.csv (601.2 MB)

Elapsed time: 00:06:32.975
Rule Parse Processing Time: 00:00:01.150
Analysis Processing Time: 00:06:23.481
Output Processing Time: 00:00:08.343

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     4.0 GiB     4.0 GiB     0           4.0 GiB
 committed:     1.0 GiB     4.0 GiB   207.5 GiB  -203.5 GiB                          ok
     reset:     0
    purged:    22.5 GiB
   touched:    64.2 KiB     8.7 MiB    53.0 GiB   -53.0 GiB                          ok
  segments:    16         140         132           8                                not all freed!
-abandoned:     0           0           0           0                                ok
   -cached:     0           0           0           0                                ok
     pages:     0           0         728.6 Ki   -728.6 Ki                           ok
-abandoned:     0           0           0           0                                ok
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:     0
    resets:     0
    purges:    12.8 Ki
   threads:    16          16           0          16                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:   392.980 s
   process: user: 2108.242 s, system: 26.089 s, faults: 17, rss: 2.8 GiB, commit: 1.0 GiB

@YamatoSecurity
Copy link
Collaborator

@fukusuket Thank you so much for the quick PR!
It did improve but still not as fast as 2.5.1.
I updated the benchmarks here: https://docs.google.com/spreadsheets/d/1XCEkJHYA5N8b7O2KldGp7q0C7_F1HFnibz7HwI90-s0/edit#gid=0
On 32GB of data, output processing time is:

2.5.1: 11 min
main branch: 1 hour 7 min
this PR: 37 min

By the way, is the rule author list only needed for outputting author names next to rules in the HTML report?
If this is the case, can we just compute that information when -H is specified?

@fukusuket
Copy link
Collaborator Author

@YamatoSecurity
Thank you so much for the benchmark! I see...🤔 it seems there is still room for improvement. I will investigate further!💪

By the way, is the rule author list only needed for outputting author names next to rules in the HTML report?

It was a necessary process even for terminal output.Therefore, it is necessary to calculate even without the -H option :(

@fukusuket
Copy link
Collaborator Author

fukusuket commented Jun 8, 2023 •
edited
Loading

@YamatoSecurity
I have one question!
ExtraFieldInfo was added in version 2.6.0, so I think you'll need to use the same profile for an accurate comparison with 2.5.1.
Is there a difference in performance even when using the same profile?

@YamatoSecurity
Copy link
Collaborator

YamatoSecurity commented Jun 8, 2023 •
edited
Loading

@fukusuket That is a good point!
Yes, -p minimal is very similar to 2.5.1's default output. With the new config file, 2.5.1 will output an extra %ExtraFieldInfo% that does not get rendered so that will add some file size. I think if the minimal profile is used for both versions it should be the same.
I will re-run my benchmark.

@codecov
Copy link

codecov bot commented Jun 8, 2023 •
edited
Loading

Codecov Report

Patch coverage: 20.00% and project coverage change: -0.01 ⚠️

Comparison is base (9a38984) 82.14% compared to head (ba791be) 82.13%.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #1090      +/-   ##
==========================================
- Coverage   82.14%   82.13%   -0.01%     
==========================================
  Files          24       24              
  Lines       19915    19918       +3     
==========================================
+ Hits        16359    16360       +1     
- Misses       3556     3558       +2
Impacted Files Coverage Δ
src/afterfact.rs 66.64% <20.00%> (-0.04%) ⬇️

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@fukusuket
Copy link
Collaborator Author

fukusuket commented Jun 8, 2023 •
edited
Loading

I checked with the data of all-evtx.tgz (6.1GB). The results are as follows.
It seems that it is a little slower when output?🤔 I will investigate where improving the output processing as much as I can.
hayabusa csv-timline -d all-evtx -o out.cvs --debug with 2.5.1 default profile.

Ver Elasped Time Rule Parse Time Analysis Time Output Time Memory peak
2.5.1 00:06:29.145 00:00:01.100 00:06:20.710 00:00:07.333 4.0 GiB
This PR 00:06:32.009 00:00:01.106 00:06:22.602 00:00:08.300 4.0 GiB

@YamatoSecurity
Copy link
Collaborator

@fukusuket Thanks! If it is just 1 second then that is no problem. I will check it on a larger dataset.

@fukusuket fukusuket self-assigned this Jun 8, 2023
@fukusuket fukusuket added the enhancement New feature or request label Jun 8, 2023
@fukusuket fukusuket added this to the v2.6.0 milestone Jun 8, 2023
@YamatoSecurity
Copy link
Collaborator

@fukusuket 悲報: Unfortunately there is not a big improvement for my larger dataset (32GB). Output time does decrease from 34 min 30 seconds to 34 min 10 sec, but 2.5.1 is only around 11 min.
Could you see if there is anything else that might be causing the slower times?

@hitenkoku hitenkoku force-pushed the 1088-fix-improve-speed-by-reduce-io branch 2 times, most recently from e031962 to 8689b7a Compare June 9, 2023 09:35
@hitenkoku
Copy link
Collaborator

@YamatoSecurity @fukusuket

My apologies. I tried to import commit e031962 in another branch and it failed.

Your correction has been submitted in ba791be and is unchanged.

@hitenkoku hitenkoku mentioned this pull request Jun 9, 2023
Merged
@fukusuket fukusuket mentioned this pull request Jun 10, 2023
Merged
@YamatoSecurity
Copy link
Collaborator

YamatoSecurity commented Jun 10, 2023 •
edited
Loading

@hitenkoku if this PR is included in #1089 should we close it?

@fukusuket
Copy link
Collaborator Author

Yes, it's included in #1088, so I'll close it! :)

@fukusuket fukusuket closed this Jun 10, 2023
@hitenkoku hitenkoku deleted the 1088-fix-improve-speed-by-reduce-io branch February 1, 2024 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hitenkoku hitenkoku Awaiting requested review from hitenkoku

@YamatoSecurity YamatoSecurity Awaiting requested review from YamatoSecurity

Assignees

@fukusuket fukusuket

Labels
enhancement New feature or request
Projects
None yet
Milestone
v2.6.0
Development

Successfully merging this pull request may close these issues.
3 participants
@fukusuket @YamatoSecurity @hitenkoku