Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chg: build.rs(for vc runtime) to rustflags in config.toml and replace default global memory allocator with mimalloc. #777

Merged
merged 10 commits into from
Oct 24, 2022

Conversation

fukusuket
Copy link
Collaborator

@fukusuket fukusuket commented Oct 23, 2022

What Changed

Evidence

Environment

Test1

Compare the below execution speed of ver1.7.2 and the fixed version 3 times each.
.\hayabusa.exe -d C:\tmp\hayabusa-sample-evtx -o out.csv

then execution time is as follows.

  • ver1.7.2
  1. Elapsed Time: 00:00:20.380
  2. Elapsed Time: 00:00:20.374
  3. Elapsed Time: 00:00:20.493
  • fixed version
  1. Elapsed time: 00:00:17.313
  2. Elapsed time: 00:00:17.236
  3. Elapsed time: 00:00:17.235

Test2

Compare the detection results ver1.7.2 and the fixed version.
I confirmed that the number of detections is the same.

ver1.7.2

Results Summary:

Events with hits / Total events: 19,549 / 47,458 (Data reduction: 27,909 events (58.81%))

Total | Unique detections: 32,864 | 580
Total | Unique critical detections: 47 (0.14%) | 19 (3.28%)
Total | Unique high detections: 6,222 (18.93%) | 260 (44.83%)
Total | Unique medium detections: 1,566 (4.77%) | 170 (29.31%)
Total | Unique low detections: 6,781 (20.63%) | 78 (13.45%)
Total | Unique informational detections: 18,248 (55.53%) | 53 (9.14%)

Dates with most total detections:
critical: 2019-07-19 (15), high: 2016-09-20 (3,656), medium: 2021-04-22 (186), low: 2016-09-20 (3,781), informational: 2016-08-19 (2,105)

Top 5 computers with most unique detections:
critical: MSEDGEWIN10 (6), IEWIN7 (3), FS03.offsec.lan (2), srvdefender01.offsec.lan (2), rootdc1.offsec.lan (2)
high: MSEDGEWIN10 (117), IEWIN7 (73), FS03.offsec.lan (34), fs03vuln.offsec.lan (28), IE10Win7 (23)
medium: MSEDGEWIN10 (66), IEWIN7 (40), FS03.offsec.lan (17), IE10Win7 (15), PC01.example.corp (15)
low: MSEDGEWIN10 (35), IEWIN7 (18), FS03.offsec.lan (16), fs03vuln.offsec.lan (13), IE10Win7 (11)
informational: MSEDGEWIN10 (18), IEWIN7 (17), fs01.offsec.lan (15), PC01.example.corp (13), FS03.offsec.lan (12)

╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Top critical alerts:                                              Top high alerts:                                    │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Sticky Key Like Backdoor Usage (10)                               Metasploit SMB Authentication (3,562)               │
│ Active Directory Replication from Non Machine Account (6)         Malicious Svc Possibly Installed (271)              │
│ Meterpreter or Cobalt Strike Getsystem Service Installation (6)   Susp Svc Installed (257)                            │
│ Defender Alert (Severe) (4)                                       PowerShell Scripts Installed as Services (253)      │
│ WannaCry Ransomware (4)                                           Suspicious Service Installation Script (250)        │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts:                                                Top low alerts:                                     │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Potentially Malicious PwSh (235)                                  Logon Failure (Wrong Password) (3,564)              │
│ Proc Injection (104)                                              Susp CmdLine (Possible LOLBIN) (1,418)              │
│ Reg Key Value Set (Sysmon Alert) (103)                            Non Interactive PowerShell (325)                    │
│ Suspicious Remote Thread Target (93)                              Rare Service Installations (321)                    │
│ Wscript Execution from Non C Drive (61)                           Windows Processes Suspicious Parent Directory (282) │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                                                                                             │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Proc Exec (11,173)                                                Explicit Logon (342)                                │
│ NetShare File Access (2,564)                                      Svc Installed (331)                                 │
│ PwSh Scriptblock (789)                                            New Non-USB PnP Device (268)                        │
│ PwSh Pipeline Exec (680)                                          Logon (Type 3 Network) (228)                        │
│ NetShare Access (433)                                             File Created (210)                                  │

fixed version

Results Summary:

Events with hits / Total events: 19,549 / 47,458 (Data reduction: 27,909 events (58.81%))

Total | Unique detections: 32,864 | 580
Total | Unique critical detections: 47 (0.14%) | 19 (3.28%)
Total | Unique high detections: 6,222 (18.93%) | 260 (44.83%)
Total | Unique medium detections: 1,566 (4.77%) | 170 (29.31%)
Total | Unique low detections: 6,781 (20.63%) | 78 (13.45%)
Total | Unique informational detections: 18,248 (55.53%) | 53 (9.14%)

Dates with most total detections:
critical: 2019-07-19 (15), high: 2016-09-20 (3,656), medium: 2021-04-22 (186), low: 2016-09-20 (3,781), informational: 2016-08-19 (2,105)

Top 5 computers with most unique detections:
critical: MSEDGEWIN10 (6), IEWIN7 (3), srvdefender01.offsec.lan (2), rootdc1.offsec.lan (2), FS03.offsec.lan (2)
high: MSEDGEWIN10 (117), IEWIN7 (73), FS03.offsec.lan (34), fs03vuln.offsec.lan (28), IE10Win7 (23)
medium: MSEDGEWIN10 (66), IEWIN7 (40), FS03.offsec.lan (17), PC01.example.corp (15), IE10Win7 (15)
low: MSEDGEWIN10 (35), IEWIN7 (18), FS03.offsec.lan (16), fs03vuln.offsec.lan (13), fs01.offsec.lan (11)
informational: MSEDGEWIN10 (18), IEWIN7 (17), fs01.offsec.lan (15), PC01.example.corp (13), IE10Win7 (12)

╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Top critical alerts:                                              Top high alerts:                                    │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Sticky Key Like Backdoor Usage (10)                               Metasploit SMB Authentication (3,562)               │
│ Meterpreter or Cobalt Strike Getsystem Service Installation (6)   Malicious Svc Possibly Installed (271)              │
│ Active Directory Replication from Non Machine Account (6)         Susp Svc Installed (257)                            │
│ Defender Alert (Severe) (4)                                       PowerShell Scripts Installed as Services (253)      │
│ WannaCry Ransomware (4)                                           Suspicious Service Installation Script (250)        │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts:                                                Top low alerts:                                     │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Potentially Malicious PwSh (235)                                  Logon Failure (Wrong Password) (3,564)              │
│ Proc Injection (104)                                              Susp CmdLine (Possible LOLBIN) (1,418)              │
│ Reg Key Value Set (Sysmon Alert) (103)                            Non Interactive PowerShell (325)                    │
│ Suspicious Remote Thread Target (93)                              Rare Service Installations (321)                    │
│ Wscript Execution from Non C Drive (61)                           Windows Processes Suspicious Parent Directory (282) │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                                                                                             │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Proc Exec (11,173)                                                Explicit Logon (342)                                │
│ NetShare File Access (2,564)                                      Svc Installed (331)                                 │
│ PwSh Scriptblock (789)                                            New Non-USB PnP Device (268)                        │
│ PwSh Pipeline Exec (680)                                          Logon (Type 3 Network) (228)                        │
│ NetShare Access (433)                                             File Created (210)                                  │

I would appreciate it if you could review🙏

closes #657

@fukusuket fukusuket changed the title chg: buildrs(for vc runtime) to rustflags in config.toml and replace global memory allocator to mimalloc. chg: build.rs(for vc runtime) to rustflags in config.toml and replace global memory allocator to mimalloc. Oct 23, 2022
@fukusuket fukusuket changed the title chg: build.rs(for vc runtime) to rustflags in config.toml and replace global memory allocator to mimalloc. chg: build.rs(for vc runtime) to rustflags in config.toml and replace global memory allocator with mimalloc. Oct 23, 2022
@fukusuket fukusuket changed the title chg: build.rs(for vc runtime) to rustflags in config.toml and replace global memory allocator with mimalloc. chg: build.rs(for vc runtime) to rustflags in config.toml and replace default global memory allocator with mimalloc. Oct 23, 2022
@hitenkoku hitenkoku self-requested a review October 23, 2022 06:40
@hitenkoku hitenkoku added this to the v1.8.0 milestone Oct 23, 2022
Copy link
Collaborator

@hitenkoku hitenkoku left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fukusuket Thank you for your pull request!
This improve is great.
LGTM

@fukusuket
Copy link
Collaborator Author

fukusuket commented Oct 23, 2022

I also compared the speed on macOS. There seems to be no regression, but no improvement either :(

Environment

  • OS: macOS montery version 12.6
  • Hard: Macbook Air(M1, 2020) , Memory 8GB, Core 8
  • Data: hayabusa-sample-evtx
  • ./hayabusa -d hayabusa-sample-evtx -o out.csv

Speed test results.

  • ver1.7.2
    Elapsed Time: 00:00:11.764
    Elapsed Time: 00:00:11.546
    Elapsed Time: 00:00:11.629

  • fixed version
    Elapsed time: 00:00:11.281
    Elapsed time: 00:00:11.889
    Elapsed time: 00:00:11.482

@fukusuket
Copy link
Collaborator Author

Unfortunately in github Actions build (windows-latest, i686-pc-windows-msvc, false) failed😭
I will check it ...

@YamatoSecurity
Copy link
Collaborator

@fukusuket sometimes cross compiling will fix it, so I set it to true.

@YamatoSecurity
Copy link
Collaborator

@fukusuket おっと。。(T_T) 残念ながらcross: trueでも駄目でした。。

@fukusuket
Copy link
Collaborator Author

ありがとうございます!🙇 crossオプションでもダメなのですね...涙 こちらでももう少し調査いたします!

@YamatoSecurity
Copy link
Collaborator

@fukusuket Now we are using mimalloc = { version = "*", default-features = false } turning off the default secure mode for better performance but even when I compiled with mimalloc = { version = "*"} I didn't notice a slowdown. Can you check if this is the same for you? If there is no performance slowdown, I would like to use the default secure mode.

@YamatoSecurity
Copy link
Collaborator

@fukusuket Also, did you check that this correctly gets statically compiled and that it will run on systems without the VC++ Redistribution package?

@fukusuket
Copy link
Collaborator Author

@YamatoSecurity
I forgot the essential test in an environment without VC++ Redistribution package...😂
I'll check it along with the mimalloc = { version = "*"} test.

@fukusuket
Copy link
Collaborator Author

fukusuket commented Oct 23, 2022

Evidence
I confirmed that running on systems without the VC++ Redistribution package.

Environment

  • OS: Windows 11 Home edition (on m1 mac parallels desktop)
  • VC++ Redistribution package not installed.

Test3

reproduce bug in v.142 as follows.
v142-error

fix version help command seccuess as follows.
v180dev-sucess

fix version live-analysis command seccuess as follows.
v180dev-l-sucess-2

@fukusuket
Copy link
Collaborator Author

fukusuket commented Oct 23, 2022

@YamatoSecurity

Evidence
I compared the performance as follows. mimalloc = { version = "*"} is slightly slower ... :(
(But it's improved, so I think it's okay mimalloc = { version = "*"} )

Environment

Test4

Compare the below execution speed(3 times each).
.\hayabusa.exe -d C:\tmp\hayabusa-sample-evtx -o out.csv

v172

  1. Elapsed Time: 00:00:21.128
  2. Elapsed Time: 00:00:20.566
  3. Elapsed Time: 00:00:20.563

mimalloc = { version = "*"}

  1. Elapsed time: 00:00:18.549
  2. Elapsed time: 00:00:18.590
  3. Elapsed time: 00:00:18.571

mimalloc = { version = "*", default-features = false }

  1. Elapsed time: 00:00:17.363
  2. Elapsed time: 00:00:17.330
  3. Elapsed time: 00:00:17.217

@fukusuket
Copy link
Collaborator Author

fukusuket commented Oct 23, 2022

Evidence
I confirmed that the 32bit exe build succeeds as follows.

Environment

  • OS: Windows 10 Home edition

Test5

i686 build succeeds as follows.

PS C:\Users\fukus\IdeaProjects\hayabusa> cargo clean                                                 
PS C:\Users\fukus\IdeaProjects\hayabusa> rustup run stable-i686-pc-windows-msvc cargo build --release
   Compiling autocfg v1.1.0      
   Compiling cfg-if v1.0.0       
   ...
   Compiling hayabusa v1.8.0-dev (C:\Users\fukus\IdeaProjects\hayabusa)
   Finished release [optimized] target(s) in 4m 28s

and exe is for 32bit
x86

Sorry, Because there is no 32-bit environment, I couldn't check the actual execution...🙇

@fukusuket
Copy link
Collaborator Author

fukusuket commented Oct 23, 2022

Evidence
I confirmed that the 32bit exe speed has not decreased.
Build with the architecture check disabled and I tested as follows.

Environment

Test6

Compare the below execution speed(3 times each).
.\hayabusa.exe -d C:\tmp\hayabusa-sample-evtx -o out.csv

commit(before fix): 96cabd6a
Elapsed time: 00:00:25.062
Elapsed time: 00:00:25.127
Elapsed time: 00:00:25.181

commit(after fix): eae32e51
Elapsed time: 00:00:22.333
Elapsed time: 00:00:22.278
Elapsed time: 00:00:22.348

@fukusuket
Copy link
Collaborator Author

Testing in my environment is complete. I would appreciate it if you could review🙏

Copy link
Collaborator

@hitenkoku hitenkoku left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. I checked added code.
LGTM

@hitenkoku hitenkoku added the enhancement New feature or request label Oct 24, 2022
Copy link
Collaborator

@YamatoSecurity YamatoSecurity left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mimallocの設定はとても悩ましいですが、取り敢えずパフォーマンスを優先したいので、default-featuresをfalseに変えました。
PRありがとうございました!

@fukusuket
Copy link
Collaborator Author

fukusuket commented Oct 24, 2022

パフォーマンス優先とのこと承知いたしました! 
確認したところ、本家のREADMEでも、セキュリティ有効化すると10%くらいは性能落ちると記載されていて、手元のテスト結果とだいたい一致しそうでした。
https:/microsoft/mimalloc
secure: mimalloc can be built in secure mode, adding guard pages, randomized allocation, encrypted free lists, etc. to protect against various heap vulnerabilities. The performance penalty is usually around 10% on average over our benchmarks.

@YamatoSecurity YamatoSecurity merged commit 012cf33 into main Oct 24, 2022
@fukusuket fukusuket deleted the chg-vc-runtime-buildrs-to-compiler-option branch October 24, 2022 03:35
@fukusuket
Copy link
Collaborator Author

Thank you so much for review and advice :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

remove build.rs & change allocator to mimalloc
3 participants