v3 candidate

This commit is contained in:
Andy Dixon
2026-06-16 11:51:07 +01:00
parent fcb19c96c8
commit 493bfed599
91 changed files with 28824 additions and 80 deletions
+277
View File
@@ -0,0 +1,277 @@
# binsnoop
A pure-Go binary analysis library for detecting and extracting embedded files, firmware images, compressed data, filesystems, and more from binary blobs. Ported from the [binwalk](https://github.com/ReFirmLabs/binwalk) Rust codebase with like-for-like scanning logic.
**Zero external dependencies** — everything is self-contained, including the Aho-Corasick multi-pattern matcher, CRC32 validation, and binary structure parsing.
## Features
- **111 built-in signatures** covering archives, compression, filesystems, images, executables, firmware headers, cryptographic constructs, and more
- **Aho-Corasick** multi-pattern scanning with overlapping match support
- **Signature validation** — every magic byte match is verified by a dedicated parser that checks headers, checksums, and structural integrity
- **Confidence scoring** — results are ranked Low / Medium / High so callers can filter noise
- **Extraction support** — carve identified objects out of a binary, or invoke external tools (7z, unsquashfs, etc.) to fully extract them
- **Filtering** — include/exclude signatures by name, restrict short signatures to offset 0, or add custom signatures
- **JSON-friendly** — all result types carry `json` struct tags for easy serialisation
## Installation
```bash
go get binsnoop
```
Or add to your `go.mod`:
```
require binsnoop v0.0.0
```
## Quick Start
```go
package main
import (
"fmt"
"binsnoop"
)
func main() {
scanner := binsnoop.New()
// Scan a file
results, err := scanner.ScanFile("firmware.bin")
if err != nil {
panic(err)
}
for _, r := range results {
fmt.Printf("[0x%08X] %s (confidence: %d, size: %d)\n",
r.Offset, r.Description, r.Confidence, r.Size)
}
}
```
## API Reference
### Creating a Scanner
```go
// Default scanner with all 111 signatures
scanner := binsnoop.New()
// Scanner with options
scanner, err := binsnoop.NewWithOptions(binsnoop.Options{
Include: []string{"gzip", "squashfs", "uimage"}, // only these signatures
Exclude: []string{"copyright"}, // remove these signatures
FullSearch: true, // match short sigs at any offset
AdditionalSignatures: []binsnoop.Signature{...}, // add custom signatures
})
```
### Scanning
```go
// Scan raw bytes
results := scanner.ScanBytes(data)
// Scan a file on disk
results, err := scanner.ScanFile("/path/to/firmware.bin")
```
Both return `[]SignatureResult` sorted by offset.
### Analysis (Scan + Extraction)
```go
// Analyse raw bytes (with optional extraction)
analysis := scanner.AnalyzeBytes(data, "firmware.bin", true)
// Analyse a file on disk
analysis, err := scanner.AnalyzeFile("/path/to/firmware.bin", true)
```
Returns `*AnalysisResults` containing the file map and extraction outcomes.
### Carving
Extract a single identified object from raw data:
```go
err := binsnoop.CarveResult(data, results[0], "/tmp/output")
```
This writes the bytes at the result's offset/size to a file in the output directory.
### Listing Signatures
```go
names := binsnoop.ListSignatures() // []string of all registered signature names
```
### Scanner Info
```go
scanner.SignatureCount() // number of registered signatures (111)
scanner.PatternCount() // total magic byte patterns across all signatures
```
## Types
### `SignatureResult`
```go
type SignatureResult struct {
Offset int `json:"offset"` // byte offset in the input data
ID string `json:"id"` // unique identifier for this match
Size int `json:"size"` // size in bytes (0 = unknown)
Name string `json:"name"` // signature name (e.g. "gzip")
Confidence uint8 `json:"confidence"` // 0=Low, 128=Medium, 250=High
Description string `json:"description"` // human-readable description
AlwaysDisplay bool `json:"always_display"` // show even if low confidence
ExtractionDeclined bool `json:"extraction_declined"` // parser advises against extraction
PreferredExtractor *Extractor `json:"-"` // extractor to use for this result
}
```
### `Options`
```go
type Options struct {
Include []string // restrict to these signature names
Exclude []string // remove these signature names
FullSearch bool // short sigs match at any offset
AdditionalSignatures []Signature // user-defined signatures
}
```
### `AnalysisResults`
```go
type AnalysisResults struct {
FilePath string `json:"file_path"`
FileMap []SignatureResult `json:"file_map"`
Extractions map[string]*ExtractionResult `json:"extractions"`
}
```
### Confidence Levels
| Constant | Value | Meaning |
|----------|-------|---------|
| `ConfidenceLow` | 0 | Magic matched but weak validation |
| `ConfidenceMedium` | 128 | Header parsed and mostly valid |
| `ConfidenceHigh` | 250 | Full structural validation passed |
## Custom Signatures
You can add your own signatures at scanner creation:
```go
mySignature := binsnoop.Signature{
Name: "my_format",
Magic: [][]byte{[]byte("MYMAGIC")},
Parser: func(data []byte, offset int) (*binsnoop.SignatureResult, error) {
// Validate the match and return a result
if offset+16 > len(data) {
return nil, fmt.Errorf("insufficient data")
}
return &binsnoop.SignatureResult{
Offset: offset,
Size: 1024,
Description: "My custom format, size: 1024 bytes",
Confidence: binsnoop.ConfidenceHigh,
}, nil
},
Description: "My custom file format",
}
scanner, _ := binsnoop.NewWithOptions(binsnoop.Options{
AdditionalSignatures: []binsnoop.Signature{mySignature},
})
```
## Supported Signatures
### Archives (12)
gzip, zip, tarball, 7zip, rar, cab, deb, cpio, arj, iso9660, dahua\_zip
### Compression (10)
lzma, bzip2, xz, zstd, lz4, lzop, lzfse, zlib, compress'd
### Filesystems (13)
squashfs, cramfs, jffs2, yaffs, romfs, ext, fat, ntfs, apfs, btrfs, logfs, ubi, ubifs
### Images & Media (7)
png, jpeg, bmp, gif, svg, riff, pdf
### Executables (2)
elf, pe
### Firmware (32)
uimage, uboot, trx, tplink, tplink\_rtos, dlob, arcadyan, chk, seama, cfe, jboot\_arm, jboot\_stag, jboot\_sch2, packimg, binhdr, autel, android\_sparse, android\_bootimg, dtb, dlink\_tlv, dlke, shrs, dkbs, dms, encfw, encrpted\_img, wince, mh01, csman, dxbc, matter\_ota, rtk
### Cryptographic (17)
aes\_sbox, aes\_forward\_table, aes\_reverse\_table, aes\_rcon, aes\_acceleration\_table, rsa, openssl, gpg\_signed, pem\_certificate, pem\_public\_key, pem\_private\_key, pkcs\_der\_hash, luks, crc32, sha256, md5
### Miscellaneous (18)
copyright, wind\_kernel, vxworks\_symtab, ecos, linux\_kernel, linux\_boot\_image, linux\_arm\_zimage, linux\_arm64\_boot\_image, efigpt, mbr, uefi\_pi\_volume, uefi\_capsule, pchrom, pcapng, pjl, srecord, dmg, qnx\_ifs, dpapi, qcow
## Extraction
Many signatures have associated extractors that invoke external command-line tools. When you call `AnalyzeFile` or `AnalyzeBytes` with `extract: true`, binsnoop will:
1. Carve the identified data to a temporary file
2. Run the configured external tool (e.g. `7z x`, `unsquashfs`, `jefferson`)
3. Return the extraction results including output directory and success/failure status
**Common external tools used:**
| Tool | Signatures |
|------|-----------|
| `7z` | zip, cpio, arj, iso9660, xz, lzma, squashfs, dmg, efigpt |
| `gzip` | gzip |
| `bzip2` | bzip2 |
| `zstd` | zstd |
| `lz4` | lz4 |
| `unsquashfs` | squashfs |
| `jefferson` | jffs2 |
| `unyaffs` | yaffs |
| `tsk_recover` | ext, fat, ntfs |
| `uefi-firmware-parser` | uefi\_pi\_volume, uefi\_capsule, pchrom |
| `simg2img` | android\_sparse |
| `dtc` | dtb |
## How It Works
### Scanning Algorithm
1. **Short-signature pass** — signatures marked `Short` (e.g. zlib, MBR) are checked only at offset 0 (unless `FullSearch` is enabled)
2. **Aho-Corasick scan** — all magic byte patterns are compiled into an Aho-Corasick automaton for efficient multi-pattern matching. When a high-confidence sized match is found, scanning restarts from the end of that match
3. **Validation** — every magic match is passed to its signature parser which validates headers, checksums, and structure
4. **Post-processing** — results are sorted by offset, overlapping/duplicate matches are resolved (higher confidence wins), and unknown sizes are estimated based on the next significant match or EOF
### Architecture
```
binsnoop.go Public API (New, ScanBytes, ScanFile, AnalyzeFile, CarveResult, etc.)
scanner.go Core scan engine (Aho-Corasick integration, post-processing)
match.go Self-contained Aho-Corasick trie with overlapping match support
parse.go Generic binary structure parser (u8/u16/u24/u32/u64, big/little endian)
types.go Core types (SignatureResult, Signature, Extractor, Options, etc.)
common.go Utility functions (CRC32, endian helpers, string helpers)
extract.go Carving and external tool execution
magic.go Signature registry — wires all 111 signatures together
sig_archives.go Archive signature parsers
sig_compression.go Compression signature parsers
sig_filesystems.go Filesystem signature parsers
sig_images.go Image/media signature parsers
sig_executables.go Executable signature parsers
sig_firmware.go Firmware header signature parsers
sig_crypto.go Cryptographic signature parsers
sig_misc.go Miscellaneous signature parsers
```
## License
This project is a Go port of [binwalk](https://github.com/ReFirmLabs/binwalk) by ReFirmLabs. See the original project for licensing details.
+177
View File
@@ -0,0 +1,177 @@
// Package binsnoop identifies and optionally extracts files embedded inside binary data.
//
// It is a Go port of the binwalk scanning engine, implementing all magic byte signatures,
// structure parsers, and the Aho-Corasick based multi-pattern matching algorithm.
//
// # Quick Start
//
// scanner := binsnoop.New()
// data, _ := os.ReadFile("firmware.bin")
// results := scanner.Scan(data)
// for _, r := range results {
// fmt.Printf("0x%08X %s\n", r.Offset, r.Description)
// }
//
// # Extraction
//
// // Extract a specific result
// err := binsnoop.CarveResult(data, results[0], "/tmp/output")
//
// # Filtering
//
// // Only scan for specific signatures
// scanner := binsnoop.NewWithOptions(Options{Include: []string{"gzip", "squashfs"}})
package binsnoop
import (
"fmt"
"os"
"path/filepath"
"strings"
)
// Options configures the Scanner.
type Options struct {
// Include restricts scanning to only these signature names.
Include []string
// Exclude removes these signature names from scanning.
Exclude []string
// FullSearch makes short signatures match at any offset, not just offset 0.
FullSearch bool
// AdditionalSignatures adds user-defined signatures to the scanner.
AdditionalSignatures []Signature
}
// New creates a Scanner with all default signatures enabled.
func New() *Scanner {
s, _ := NewWithOptions(Options{})
return s
}
// NewWithOptions creates a Scanner with the specified options.
func NewWithOptions(opts Options) (*Scanner, error) {
s := &Scanner{
patternSigTable: make(map[int]Signature),
extractorLookup: make(map[string]*Extractor),
}
allSigs := AllSignatures()
if opts.AdditionalSignatures != nil {
allSigs = append(allSigs, opts.AdditionalSignatures...)
}
for _, sig := range allSigs {
if !includeSig(&sig, opts.Include, opts.Exclude) {
continue
}
s.signatureCount++
s.patternCount += len(sig.Magic)
// Register extractor
var ext *Extractor
if sig.Extractor != nil {
extCopy := *sig.Extractor
ext = &extCopy
}
s.extractorLookup[sig.Name] = ext
for _, pattern := range sig.Magic {
if sig.Short && !opts.FullSearch {
s.shortSignatures = append(s.shortSignatures, sig)
break
}
s.patternSigTable[len(s.patterns)] = sig
s.patterns = append(s.patterns, pattern)
}
}
s.compile()
return s, nil
}
// ScanBytes scans raw bytes for signatures and returns all matches.
func (s *Scanner) ScanBytes(data []byte) []SignatureResult {
return s.Scan(data)
}
// ScanFile reads a file from disk and scans it for signatures.
func (s *Scanner) ScanFile(path string) ([]SignatureResult, error) {
data, err := ReadFile(path)
if err != nil {
return nil, fmt.Errorf("binsnoop: read file: %w", err)
}
return s.Scan(data), nil
}
// AnalyzeBytes scans and optionally extracts from raw bytes.
func (s *Scanner) AnalyzeBytes(data []byte, filePath string, extract bool) *AnalysisResults {
results := &AnalysisResults{
FilePath: filePath,
Extractions: make(map[string]*ExtractionResult),
}
results.FileMap = s.Scan(data)
if extract && len(results.FileMap) > 0 {
results.Extractions = s.Extract(data, filePath, results.FileMap)
}
return results
}
// AnalyzeFile reads a file from disk, scans, and optionally extracts.
func (s *Scanner) AnalyzeFile(path string, extract bool) (*AnalysisResults, error) {
data, err := ReadFile(path)
if err != nil {
return nil, fmt.Errorf("binsnoop: read file: %w", err)
}
return s.AnalyzeBytes(data, path, extract), nil
}
// CarveResult extracts raw bytes for a specific signature result to a file.
func CarveResult(data []byte, result SignatureResult, outputDir string) error {
if result.Offset < 0 || result.Offset+result.Size > len(data) {
return fmt.Errorf("binsnoop: result offset/size exceeds data bounds")
}
if err := os.MkdirAll(outputDir, 0755); err != nil {
return fmt.Errorf("binsnoop: mkdir: %w", err)
}
outPath := filepath.Join(outputDir, fmt.Sprintf("%x_%s.bin", result.Offset, result.Name))
return os.WriteFile(outPath, data[result.Offset:result.Offset+result.Size], 0644)
}
// SignatureCount returns the number of loaded signatures.
func (s *Scanner) SignatureCount() int { return s.signatureCount }
// PatternCount returns the total number of magic byte patterns.
func (s *Scanner) PatternCount() int { return s.patternCount }
// ListSignatures returns the names of all available signatures.
func ListSignatures() []string {
sigs := AllSignatures()
names := make([]string, len(sigs))
for i, s := range sigs {
names[i] = s.Name
}
return names
}
func includeSig(sig *Signature, include, exclude []string) bool {
if len(include) > 0 {
for _, inc := range include {
if strings.EqualFold(sig.Name, inc) {
return true
}
}
return false
}
if len(exclude) > 0 {
for _, exc := range exclude {
if strings.EqualFold(sig.Name, exc) {
return false
}
}
}
return true
}
+127
View File
@@ -0,0 +1,127 @@
package binsnoop
import (
"encoding/binary"
"fmt"
"hash/crc32"
"os"
"time"
)
const MaxFileReadBytes int64 = 64 << 20
// CRC32 calculates the CRC32 checksum using the IEEE polynomial.
func CRC32(data []byte) uint32 {
return crc32.ChecksumIEEE(data)
}
// EpochToString converts a Unix timestamp to a formatted time string.
func EpochToString(epoch uint32) string {
t := time.Unix(int64(epoch), 0).UTC()
return t.Format("2006-01-02 15:04:05")
}
// GetCString extracts a NULL-terminated C string from raw bytes.
func GetCString(data []byte) string {
for i, b := range data {
if b == 0 {
s := string(data[:i])
// Validate UTF-8 printability
for _, c := range s {
if c == 0 {
return ""
}
}
return s
}
}
return string(data)
}
// IsASCIINumber returns true if the byte is an ASCII digit (0-9).
func IsASCIINumber(b byte) bool {
return b >= '0' && b <= '9'
}
// IsPrintableASCII returns true if the byte is a printable ASCII character.
func IsPrintableASCII(b byte) bool {
return b >= 0x0A && b <= 0x7E
}
// IsOffsetSafe validates that the next offset is within bounds and progressing forward.
func IsOffsetSafe(availableData, nextOffset int, lastOffset *int) bool {
if lastOffset != nil {
if *lastOffset >= nextOffset {
return false
}
}
return nextOffset < availableData
}
// ReadFile reads a bounded file into memory for scanner APIs that require a
// byte slice. Larger callers should use a streaming document path.
func ReadFile(path string) ([]byte, error) {
return ReadFileLimited(path, MaxFileReadBytes)
}
func ReadFileLimited(path string, maxBytes int64) ([]byte, error) {
info, err := os.Stat(path)
if err != nil {
return nil, err
}
if info.IsDir() {
return nil, fmt.Errorf("binsnoop: path is a directory")
}
if maxBytes > 0 && info.Size() > maxBytes {
return nil, fmt.Errorf("binsnoop: file is too large: %d bytes exceeds %d bytes", info.Size(), maxBytes)
}
return os.ReadFile(path)
}
// LE16 reads a little-endian uint16 from data at the given offset.
func LE16(data []byte, offset int) uint16 {
if offset+2 > len(data) {
return 0
}
return binary.LittleEndian.Uint16(data[offset:])
}
// BE16 reads a big-endian uint16 from data at the given offset.
func BE16(data []byte, offset int) uint16 {
if offset+2 > len(data) {
return 0
}
return binary.BigEndian.Uint16(data[offset:])
}
// LE32 reads a little-endian uint32 from data at the given offset.
func LE32(data []byte, offset int) uint32 {
if offset+4 > len(data) {
return 0
}
return binary.LittleEndian.Uint32(data[offset:])
}
// BE32 reads a big-endian uint32 from data at the given offset.
func BE32(data []byte, offset int) uint32 {
if offset+4 > len(data) {
return 0
}
return binary.BigEndian.Uint32(data[offset:])
}
// LE64 reads a little-endian uint64 from data at the given offset.
func LE64(data []byte, offset int) uint64 {
if offset+8 > len(data) {
return 0
}
return binary.LittleEndian.Uint64(data[offset:])
}
// BE64 reads a big-endian uint64 from data at the given offset.
func BE64(data []byte, offset int) uint64 {
if offset+8 > len(data) {
return 0
}
return binary.BigEndian.Uint64(data[offset:])
}
@@ -0,0 +1,22 @@
package binsnoop
import (
"os"
"path/filepath"
"strings"
"testing"
)
func TestReadFileLimitedRejectsOversizedFile(t *testing.T) {
path := filepath.Join(t.TempDir(), "firmware.bin")
if err := os.WriteFile(path, []byte("abcd"), 0600); err != nil {
t.Fatalf("WriteFile() failed: %v", err)
}
_, err := ReadFileLimited(path, 3)
if err == nil {
t.Fatal("expected oversized file to be rejected")
}
if !strings.Contains(err.Error(), "file is too large") {
t.Fatalf("unexpected error: %v", err)
}
}
+124
View File
@@ -0,0 +1,124 @@
package binsnoop
import (
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
)
// Carve writes a slice of data to a file on disk.
func Carve(outputPath string, data []byte, offset, size int) error {
if offset < 0 || offset+size > len(data) {
return fmt.Errorf("carve: offset %d + size %d exceeds data length %d", offset, size, len(data))
}
dir := filepath.Dir(outputPath)
if err := os.MkdirAll(dir, 0755); err != nil {
return fmt.Errorf("carve: mkdir: %w", err)
}
return os.WriteFile(outputPath, data[offset:offset+size], 0644)
}
// executeExternal runs an external extraction tool.
func executeExternal(ext *Extractor, carvedFilePath, outputDir string) *ExtractionResult {
result := &ExtractionResult{
ExtractorName: ext.Command,
OutputDirectory: outputDir,
}
if err := os.MkdirAll(outputDir, 0755); err != nil {
return result
}
// Build arguments, replacing the source file placeholder
args := make([]string, len(ext.Arguments))
for i, arg := range ext.Arguments {
args[i] = strings.ReplaceAll(arg, SourceFilePlaceholder, carvedFilePath)
}
cmd := exec.Command(ext.Command, args...)
cmd.Dir = outputDir
output, err := cmd.CombinedOutput()
_ = output
exitCode := 0
if err != nil {
if exitErr, ok := err.(*exec.ExitError); ok {
exitCode = exitErr.ExitCode()
} else {
return result
}
}
// Check if exit code is in the accepted list
for _, code := range ext.ExitCodes {
if exitCode == code {
result.Success = true
break
}
}
result.DoNotRecurse = ext.DoNotRecurse
return result
}
// executeInternal runs an internal extraction function.
func executeInternal(ext *Extractor, data []byte, offset int, outputDir string) *ExtractionResult {
if ext.Function == nil {
return &ExtractionResult{}
}
result := ext.Function(data, offset, outputDir)
if result == nil {
result = &ExtractionResult{}
}
result.DoNotRecurse = ext.DoNotRecurse
return result
}
// ExecuteExtraction performs extraction for a signature result.
func ExecuteExtraction(data []byte, filePath string, sig *SignatureResult, ext *Extractor) *ExtractionResult {
if ext == nil {
return &ExtractionResult{}
}
// Use preferred extractor if specified
actualExt := ext
if sig.PreferredExtractor != nil {
actualExt = sig.PreferredExtractor
}
// Build output directory path
baseName := filepath.Base(filePath)
outputDir := filepath.Join(filepath.Dir(filePath), baseName+".extracted",
fmt.Sprintf("%x", sig.Offset))
switch actualExt.Type {
case ExtractorTypeInternal:
return executeInternal(actualExt, data, sig.Offset, outputDir)
case ExtractorTypeExternal:
// Carve data to a temp file for external tools
if err := os.MkdirAll(outputDir, 0755); err != nil {
return &ExtractionResult{OutputDirectory: outputDir}
}
carvedFile := filepath.Join(outputDir, fmt.Sprintf("carved.%s", actualExt.Extension))
endOffset := sig.Offset + sig.Size
if endOffset > len(data) {
endOffset = len(data)
}
if err := os.WriteFile(carvedFile, data[sig.Offset:endOffset], 0644); err != nil {
return &ExtractionResult{OutputDirectory: outputDir}
}
result := executeExternal(actualExt, carvedFile, outputDir)
result.OutputDirectory = outputDir
result.ExtractorName = actualExt.Command
return result
default:
return &ExtractionResult{}
}
}
+268
View File
@@ -0,0 +1,268 @@
package binsnoop
// AllSignatures returns every supported signature definition.
// This is the equivalent of magic::patterns() in the Rust binwalk codebase.
func AllSignatures() []Signature {
return []Signature{
// ===================== Archives =====================
{Name: "gzip", Magic: gzipMagic(), Parser: gzipParser, Description: gzipDescription, Extractor: extGzip()},
{Name: "deb", Magic: debMagic(), Parser: debParser, Description: debDescription},
{Name: "7zip", Magic: sevenzipMagic(), Parser: sevenzipParser, Description: sevenzipDescription, Extractor: extSevenzip()},
{Name: "xz", Magic: XZMagic(), Parser: XZParser, Description: XZDescription, Extractor: extLzma()},
{Name: "tarball", Magic: tarballMagic(), Parser: tarballParser, Description: tarballDescription, Extractor: extTar()},
{Name: "zip", Magic: zipMagic(), Parser: zipParser, Description: zipDescription, Extractor: extSevenzip()},
{Name: "cpio", Magic: cpioMagic(), Parser: cpioParser, Description: cpioDescription, Extractor: extSevenzip()},
{Name: "iso9660", Magic: iso9660Magic(), Parser: iso9660Parser, Description: iso9660Description, Extractor: extSevenzip()},
{Name: "rar", Magic: rarMagic(), Parser: rarParser, Description: rarDescription, Extractor: extRar()},
{Name: "cab", Magic: cabMagic(), Parser: cabParser, Description: cabDescription, Extractor: extCab()},
{Name: "arj", Magic: arjMagic(), Parser: arjParser, Description: arjDescription, Extractor: extSevenzip()},
{Name: "dahua_zip", Magic: dahuaZipMagic(), Parser: dahuaZipParser, Description: dahuaZipDescription, Extractor: extSevenzip()},
// ===================== Compression =====================
{Name: "lzma", Magic: LZMAMagic(), Parser: LZMAParser, Description: LZMADescription, Extractor: extLzma()},
{Name: "bzip2", Magic: Bzip2Magic(), Parser: Bzip2Parser, Description: Bzip2Description, Extractor: extBzip2()},
{Name: "zstd", Magic: ZSTDMagic(), Parser: ZSTDParser, Description: ZSTDDescription, Extractor: extZstd()},
{Name: "lz4", Magic: LZ4Magic(), Parser: LZ4Parser, Description: LZ4Description, Extractor: extLz4()},
{Name: "lzop", Magic: LZOPMagic(), Parser: LZOPParser, Description: LZOPDescription, Extractor: extLzop()},
{Name: "lzfse", Magic: LZFSEMagic(), Parser: LZFSEParser, Description: LZFSEDescription, Extractor: extLzfse()},
{Name: "zlib", Short: true, Magic: ZlibMagic(), Parser: ZlibParser, Description: ZlibDescription, Extractor: extZlib()},
{Name: "compressd", Short: true, Magic: CompressdMagic(), Parser: CompressdParser, Description: CompressdDescription, Extractor: extSevenzip()},
// ===================== Filesystems =====================
{Name: "squashfs", Magic: squashfs_magic(), Parser: squashfs_parser, Description: SquashFSDescription, Extractor: extSquashfs()},
{Name: "cramfs", Magic: cramfs_magic(), Parser: cramfs_parser, Description: CramFSDescription, Extractor: extSevenzip()},
{Name: "jffs2", Magic: jffs2_magic(), Parser: jffs2_parser, Description: JFFS2Description, Extractor: extJffs2()},
{Name: "yaffs", Magic: yaffs_magic(), Parser: yaffs_parser, Description: YAFFSDescription, Extractor: extYaffs2()},
{Name: "romfs", Magic: romfs_magic(), Parser: romfs_parser, Description: RomFSDescription, Extractor: extRomfs()},
{Name: "ext", Magic: ext_magic(), Parser: ext_parser, Description: EXTDescription, Extractor: extTsk()},
{Name: "fat", Short: true, MagicOffset: FATMagicOffset, Magic: fat_magic(), Parser: fat_parser, Description: FATDescription, Extractor: extTsk()},
{Name: "ntfs", Magic: ntfs_magic(), Parser: ntfs_parser, Description: NTFSDescription, Extractor: extTsk()},
{Name: "apfs", Magic: apfs_magic(), Parser: apfs_parser, Description: APFSDescription, Extractor: extSevenzip()},
{Name: "btrfs", AlwaysDisplay: true, Magic: btrfs_magic(), Parser: btrfs_parser, Description: BTRFSDescription},
{Name: "logfs", Magic: logfs_magic(), Parser: logfs_parser, Description: LogFSDescription},
{Name: "ubi", Magic: ubi_magic(), Parser: ubi_parser, Description: UBIImageDescription, Extractor: extUbi()},
{Name: "ubifs", Magic: ubifs_magic(), Parser: ubifs_parser, Description: UBIFSDescription, Extractor: extUbifs()},
// ===================== Images & Media =====================
{Name: "png", Magic: PNGMagic(), Parser: PNGParser, Description: PNGDescription, Extractor: extPng()},
{Name: "jpeg", Magic: JPEGMagic(), Parser: JPEGParser, Description: JPEGDescription, Extractor: extJpeg()},
{Name: "bmp", Magic: BMPMagic(), Parser: BMPParser, Description: BMPDescription, Extractor: extBmp()},
{Name: "gif", Magic: GIFMagic(), Parser: GIFParser, Description: GIFDescription, Extractor: extGif()},
{Name: "svg", Magic: SVGMagic(), Parser: SVGParser, Description: SVGDescription, Extractor: extSvg()},
{Name: "riff", Magic: RIFFMagic(), Parser: RIFFParser, Description: RIFFDescription, Extractor: extRiff()},
{Name: "pdf", Magic: PDFMagic(), Parser: PDFParser, Description: PDFDescription},
// ===================== Executables =====================
{Name: "elf", Magic: ELFMagic(), Parser: ELFParser, Description: ELFDescription},
{Name: "pe", Magic: PEMagic(), Parser: PEParser, Description: PEDescription},
// ===================== Firmware Headers =====================
{Name: "uimage", Magic: uimageMagic(), Parser: uimageParser, Description: uimageDescription, Extractor: extUimage()},
{Name: "packimg", Magic: packimgMagic(), Parser: packimgParser, Description: packimgDescription},
{Name: "uboot", AlwaysDisplay: true, Magic: ubootMagic(), Parser: ubootParser, Description: ubootDescription},
{Name: "trx", Magic: trxMagic(), Parser: trxParser, Description: trxDescription, Extractor: extTrx()},
{Name: "tplink", Magic: tplinkMagic(), Parser: tplinkParser, Description: tplinkDescription},
{Name: "tplink_rtos", Magic: tplinkRTOSMagic(), Parser: tplinkRTOSParser, Description: tplinkRTOSDescription},
{Name: "dlob", Magic: dlobMagic(), Parser: dlobParser, Description: dlobDescription},
{Name: "arcadyan", Magic: arcadyanMagic(), Parser: arcadyanParser, Description: arcadyanDescription},
{Name: "chk", Magic: chkMagic(), Parser: chkParser, Description: chkDescription},
{Name: "seama", AlwaysDisplay: true, Magic: seamaMagic(), Parser: seamaParser, Description: seamaDescription},
{Name: "cfe", AlwaysDisplay: true, Magic: cfeMagic(), Parser: cfeParser, Description: cfeDescription},
{Name: "jboot_arm", Magic: jbootARMMagic(), Parser: jbootARMParser, Description: jbootARMDescription},
{Name: "jboot_stag", Magic: jbootSTAGMagic(), Parser: jbootSTAGParser, Description: jbootSTAGDescription},
{Name: "jboot_sch2", Magic: jbootSCH2Magic(), Parser: jbootSCH2Parser, Description: jbootSCH2Description, Extractor: extJboot()},
{Name: "binhdr", Magic: binhdrMagic(), Parser: binhdrParser, Description: binhdrDescription},
{Name: "autel", Magic: autelMagic(), Parser: autelParser, Description: autelDescription, Extractor: extAutel()},
{Name: "android_sparse", Magic: androidSparseMagic(), Parser: androidSparseParser, Description: androidSparseDescription, Extractor: extAndroidSparse()},
{Name: "dtb", Magic: dtbMagic(), Parser: dtbParser, Description: dtbDescription, Extractor: extDtb()},
{Name: "android_bootimg", Magic: androidBootimgMagic(), Parser: androidBootimgParser, Description: androidBootimgDescription},
{Name: "dlink_tlv", Magic: dlinkTLVMagic(), Parser: dlinkTLVParser, Description: dlinkTLVDescription, Extractor: extEncfw()},
{Name: "dlke", Magic: dlkeMagic(), Parser: dlkeParser, Description: dlkeDescription, Extractor: extEncfw()},
{Name: "shrs", Magic: shrsMagic(), Parser: shrsParser, Description: shrsDescription, Extractor: extEncfw()},
{Name: "dkbs", Magic: dkbsMagic(), Parser: dkbsParser, Description: dkbsDescription, Extractor: extEncfw()},
{Name: "dms", Magic: dmsMagic(), Parser: dmsParser, Description: dmsDescription},
{Name: "encfw", Short: true, AlwaysDisplay: true, Magic: encfwMagic(), Parser: encfwParser, Description: encfwDescription, Extractor: extEncfw()},
{Name: "encrpted_img", Short: true, Magic: encrptedImgMagic(), Parser: encrptedImgParser, Description: encrptedImgDescription, Extractor: extEncfw()},
{Name: "wince", Magic: winceMagic(), Parser: winceParser, Description: winceDescription},
{Name: "mh01", Magic: mh01Magic(), Parser: mh01Parser, Description: mh01Description},
{Name: "csman", Short: true, Magic: csmanMagic(), Parser: csmanParser, Description: csmanDescription},
{Name: "dxbc", Magic: dxbcMagic(), Parser: dxbcParser, Description: dxbcDescription},
{Name: "matter_ota", Short: true, Magic: matterOTAMagic(), Parser: matterOTAParser, Description: matterOTADescription},
{Name: "rtk", Short: true, Magic: rtkMagic(), Parser: rtkParser, Description: rtkDescription},
{Name: "dpapi", Short: true, AlwaysDisplay: true, Magic: dpapiMagic(), Parser: dpapiParser, Description: dpapiDescription},
{Name: "qcow", Short: true, AlwaysDisplay: true, Magic: qcowMagic(), Parser: qcowParser, Description: qcowDescription},
// ===================== Crypto & Hashes =====================
{Name: "aes_sbox", Magic: aesSBoxMagic(), Parser: aesSBoxParser, Description: DescriptionAESSBox},
{Name: "aes_forward_table", Magic: aesForwardTableMagic(), Parser: aesForwardTableParser, Description: DescriptionAESFT},
{Name: "aes_reverse_table", Magic: aesReverseTableMagic(), Parser: aesReverseTableParser, Description: DescriptionAESRT},
{Name: "aes_rcon", Magic: aesRconMagic(), Parser: aesRconParser, Description: DescriptionAESRCON},
{Name: "aes_acceleration_table", Magic: aesAccelerationTableMagic(), Parser: aesAccelerationTableParser, Description: DescriptionAESAcc},
{Name: "rsa", AlwaysDisplay: true, Magic: rsaMagic(), Parser: rsaParser, Description: DescriptionRSA},
{Name: "openssl", AlwaysDisplay: true, Magic: opensslMagic(), Parser: opensslParser, Description: DescriptionOpenSSL, Extractor: extEncfw()},
{Name: "gpg_signed", Short: true, Magic: gpgSignedMagic(), Parser: gpgSignedParser, Description: DescriptionGPGSigned, Extractor: extGpg()},
{Name: "pem_certificate", Magic: pemCertificateMagic(), Parser: pemParser, Description: DescriptionPEMCertificate, Extractor: extPemCert()},
{Name: "pem_public_key", AlwaysDisplay: true, Magic: pemPublicKeyMagic(), Parser: pemParser, Description: DescriptionPEMPublicKey, Extractor: extPemKey()},
{Name: "pem_private_key", AlwaysDisplay: true, Magic: pemPrivateKeyMagic(), Parser: pemParser, Description: DescriptionPEMPrivateKey, Extractor: extPemKey()},
{Name: "pkcs_der_hash", Magic: derHashMagic(), Parser: derHashParser, Description: DescriptionDERHash},
{Name: "luks", Magic: luksMagic(), Parser: luksParser, Description: DescriptionLUKS},
{Name: "crc32", Magic: crc32ConstMagic(), Parser: crc32Parser, Description: DescriptionCRC32},
{Name: "sha256", Magic: sha256ConstMagic(), Parser: sha256Parser, Description: DescriptionSHA256},
{Name: "md5", Magic: md5ConstMagic(), Parser: md5Parser, Description: DescriptionMD5},
// ===================== Misc =====================
{Name: "copyright", Magic: copyrightMagic(), Parser: copyrightParser, Description: DescriptionCopyright},
{Name: "wind_kernel", AlwaysDisplay: true, Magic: windKernelMagic(), Parser: windKernelParser, Description: DescriptionWINDKernelVersion},
{Name: "vxworks_symtab", AlwaysDisplay: true, Magic: vxworksSymtabMagic(), Parser: vxworksSymtabParser, Description: DescriptionVxWorksSymTab},
{Name: "ecos", AlwaysDisplay: true, Magic: ecosExceptionHandlerMagic(), Parser: ecosExceptionHandlerParser, Description: DescriptionECosExceptionHandler},
{Name: "linux_kernel", AlwaysDisplay: true, Magic: linuxKernelVersionMagic(), Parser: linuxKernelVersionParser, Description: DescriptionLinuxKernelVersion, Extractor: extLinuxKernel()},
{Name: "linux_boot_image", Magic: linuxBootImageMagic(), Parser: linuxBootImageParser, Description: DescriptionLinuxBootImage},
{Name: "linux_arm_zimage", Magic: linuxARMzImageMagic(), Parser: linuxARMzImageParser, Description: DescriptionLinuxARMzImage},
{Name: "linux_arm64_boot_image", Magic: linuxARM64BootImageMagic(), Parser: linuxARM64BootImageParser, Description: DescriptionLinuxARM64BootImage},
{Name: "efigpt", Magic: efigptMagic(), Parser: efigptParser, Description: DescriptionEFIGPT, Extractor: extSevenzip()},
{Name: "mbr", Short: true, MagicOffset: MBRMagicOffset, AlwaysDisplay: true, Magic: mbrMagic(), Parser: mbrParser, Description: DescriptionMBR, Extractor: extMbr()},
{Name: "uefi_pi_volume", Magic: uefiVolumeMagic(), Parser: uefiVolumeParser, Description: DescriptionUEFIVolume, Extractor: extUefi()},
{Name: "uefi_capsule", Magic: uefiCapsuleMagic(), Parser: uefiCapsuleParser, Description: DescriptionUEFICapsule, Extractor: extUefi()},
{Name: "pchrom", Magic: pchRomMagic(), Parser: pchRomParser, Description: DescriptionPCHRom, Extractor: extUefi()},
{Name: "pcapng", Magic: pcapngMagic(), Parser: pcapngParser, Description: DescriptionPcapNG, Extractor: extPcap()},
{Name: "pjl", Magic: pjlMagic(), Parser: pjlParser, Description: DescriptionPJL},
{Name: "srecord", Magic: srecordMagic(), Parser: srecordParser, Description: DescriptionSRecord, Extractor: extSrec()},
{Name: "srecord_generic", Short: true, Magic: srecordGenericMagic(), Parser: srecordParser, Description: DescriptionSRecordGeneric, Extractor: extSrec()},
{Name: "dmg", Magic: dmgMagic(), Parser: dmgParser, Description: DescriptionDMG, Extractor: extDmg()},
{Name: "qnx_ifs", Magic: qnxIFSMagic(), Parser: qnxIFSParser, Description: DescriptionQNXIFS, Extractor: extDumpifs()},
}
}
// ============================================================================
// External extractor definitions (command-line tools)
// ============================================================================
func extSevenzip() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "7z", Extension: "bin", Arguments: []string{"x", "-y", "%e"}, ExitCodes: []int{0, 2}}
}
func extGzip() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "gzip", Extension: "gz", Arguments: []string{"-d", "-k", "-f", "%e"}, ExitCodes: []int{0}}
}
func extLzma() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "7z", Extension: "xz", Arguments: []string{"x", "-y", "%e"}, ExitCodes: []int{0, 2}}
}
func extTar() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "tar", Extension: "tar", Arguments: []string{"xf", "%e"}, ExitCodes: []int{0}}
}
func extBzip2() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "bzip2", Extension: "bz2", Arguments: []string{"-d", "-k", "-f", "%e"}, ExitCodes: []int{0}}
}
func extZstd() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "zstd", Extension: "zst", Arguments: []string{"-d", "-f", "%e"}, ExitCodes: []int{0}}
}
func extLz4() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "lz4", Extension: "lz4", Arguments: []string{"-d", "-f", "%e", "decompressed.bin"}, ExitCodes: []int{0}}
}
func extLzop() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "lzop", Extension: "lzo", Arguments: []string{"-d", "-f", "%e"}, ExitCodes: []int{0}}
}
func extLzfse() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "lzfse", Extension: "lzfse", Arguments: []string{"-decode", "-i", "%e", "-o", "decompressed.bin"}, ExitCodes: []int{0}}
}
func extZlib() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "7z", Extension: "bin", Arguments: []string{"x", "-y", "%e"}, ExitCodes: []int{0, 2}}
}
func extSquashfs() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "unsquashfs", Extension: "sqsh", Arguments: []string{"-f", "-d", "squashfs-root", "%e"}, ExitCodes: []int{0}}
}
func extJffs2() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "jefferson", Extension: "jffs2", Arguments: []string{"-d", "jffs2-root", "%e"}, ExitCodes: []int{0}}
}
func extYaffs2() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "unyaffs", Extension: "yaffs", Arguments: []string{"%e", "yaffs-root"}, ExitCodes: []int{0}}
}
func extRomfs() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "7z", Extension: "romfs", Arguments: []string{"x", "-y", "%e"}, ExitCodes: []int{0, 2}}
}
func extTsk() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "tsk_recover", Extension: "img", Arguments: []string{"-a", "-e", "%e", "tsk-root"}, ExitCodes: []int{0}}
}
func extCab() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "cabextract", Extension: "cab", Arguments: []string{"%e"}, ExitCodes: []int{0}}
}
func extRar() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "unrar", Extension: "rar", Arguments: []string{"x", "-y", "%e"}, ExitCodes: []int{0}}
}
func extUbi() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "ubireader_extract_images", Extension: "ubi", Arguments: []string{"%e"}, ExitCodes: []int{0}}
}
func extUbifs() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "ubireader_extract_files", Extension: "ubifs", Arguments: []string{"%e"}, ExitCodes: []int{0}}
}
func extPng() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "cp", Extension: "png", Arguments: []string{"%e", "image.png"}, ExitCodes: []int{0}}
}
func extJpeg() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "cp", Extension: "jpg", Arguments: []string{"%e", "image.jpg"}, ExitCodes: []int{0}}
}
func extBmp() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "cp", Extension: "bmp", Arguments: []string{"%e", "image.bmp"}, ExitCodes: []int{0}}
}
func extGif() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "cp", Extension: "gif", Arguments: []string{"%e", "image.gif"}, ExitCodes: []int{0}}
}
func extSvg() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "cp", Extension: "svg", Arguments: []string{"%e", "image.svg"}, ExitCodes: []int{0}}
}
func extRiff() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "cp", Extension: "riff", Arguments: []string{"%e", "media.riff"}, ExitCodes: []int{0}}
}
func extUimage() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "7z", Extension: "bin", Arguments: []string{"x", "-y", "%e"}, ExitCodes: []int{0, 2}}
}
func extTrx() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "7z", Extension: "bin", Arguments: []string{"x", "-y", "%e"}, ExitCodes: []int{0, 2}}
}
func extJboot() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "7z", Extension: "bin", Arguments: []string{"x", "-y", "%e"}, ExitCodes: []int{0, 2}}
}
func extAutel() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "7z", Extension: "bin", Arguments: []string{"x", "-y", "%e"}, ExitCodes: []int{0, 2}}
}
func extAndroidSparse() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "simg2img", Extension: "simg", Arguments: []string{"%e", "output.img"}, ExitCodes: []int{0}}
}
func extDtb() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "dtc", Extension: "dtb", Arguments: []string{"-I", "dtb", "-O", "dts", "-o", "device_tree.dts", "%e"}, ExitCodes: []int{0}}
}
func extEncfw() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "cp", Extension: "bin", Arguments: []string{"%e", "encrypted_firmware.bin"}, ExitCodes: []int{0}}
}
func extGpg() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "gpg", Extension: "gpg", Arguments: []string{"--decrypt", "-o", "decrypted.bin", "%e"}, ExitCodes: []int{0}}
}
func extPemCert() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "cp", Extension: "pem", Arguments: []string{"%e", "certificate.pem"}, ExitCodes: []int{0}}
}
func extPemKey() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "cp", Extension: "pem", Arguments: []string{"%e", "key.pem"}, ExitCodes: []int{0}}
}
func extLinuxKernel() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "7z", Extension: "bin", Arguments: []string{"x", "-y", "%e"}, ExitCodes: []int{0, 2}}
}
func extMbr() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "7z", Extension: "bin", Arguments: []string{"x", "-y", "%e"}, ExitCodes: []int{0, 2}}
}
func extUefi() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "uefi-firmware-parser", Extension: "bin", Arguments: []string{"-e", "-O", "%e"}, ExitCodes: []int{0}}
}
func extPcap() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "cp", Extension: "pcapng", Arguments: []string{"%e", "capture.pcapng"}, ExitCodes: []int{0}}
}
func extSrec() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "srec_cat", Extension: "srec", Arguments: []string{"%e", "-o", "output.bin", "-binary"}, ExitCodes: []int{0}}
}
func extDmg() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "7z", Extension: "dmg", Arguments: []string{"x", "-y", "%e"}, ExitCodes: []int{0, 2}}
}
func extDumpifs() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "dumpifs", Extension: "ifs", Arguments: []string{"-x", "%e"}, ExitCodes: []int{0}}
}
func extIso9660() *Extractor {
return &Extractor{Type: ExtractorTypeExternal, Command: "7z", Extension: "iso", Arguments: []string{"x", "-y", "%e"}, ExitCodes: []int{0, 2}}
}
+134
View File
@@ -0,0 +1,134 @@
package binsnoop
// ahoCorasick implements the Aho-Corasick multi-pattern byte matching algorithm.
type ahoCorasick struct {
gotoFn []map[byte]int
failFn []int
outputFn [][]int
patterns [][]byte
}
// ahoMatch represents a single pattern match.
type ahoMatch struct {
PatternIndex int
Start int
}
// newAhoCorasick builds an Aho-Corasick automaton from the given byte patterns.
func newAhoCorasick(patterns [][]byte) *ahoCorasick {
ac := &ahoCorasick{
gotoFn: []map[byte]int{make(map[byte]int)},
failFn: []int{0},
outputFn: [][]int{nil},
patterns: patterns,
}
// Build the trie (goto function)
for i, pattern := range patterns {
state := 0
for _, b := range pattern {
if next, ok := ac.gotoFn[state][b]; ok {
state = next
} else {
newState := len(ac.gotoFn)
ac.gotoFn[state][b] = newState
ac.gotoFn = append(ac.gotoFn, make(map[byte]int))
ac.failFn = append(ac.failFn, 0)
ac.outputFn = append(ac.outputFn, nil)
state = newState
}
}
ac.outputFn[state] = append(ac.outputFn[state], i)
}
// Build failure links (BFS)
queue := make([]int, 0)
for _, next := range ac.gotoFn[0] {
queue = append(queue, next)
}
for len(queue) > 0 {
r := queue[0]
queue = queue[1:]
for b, s := range ac.gotoFn[r] {
queue = append(queue, s)
state := ac.failFn[r]
for {
if _, ok := ac.gotoFn[state][b]; ok || state == 0 {
break
}
state = ac.failFn[state]
}
if next, ok := ac.gotoFn[state][b]; ok && next != s {
ac.failFn[s] = next
} else {
ac.failFn[s] = 0
}
// Merge output from failure state
if ac.outputFn[ac.failFn[s]] != nil {
ac.outputFn[s] = append(ac.outputFn[s], ac.outputFn[ac.failFn[s]]...)
}
}
}
return ac
}
// findOverlapping searches data starting at startOffset and returns all overlapping matches.
func (ac *ahoCorasick) findOverlapping(data []byte, startOffset int) []ahoMatch {
var matches []ahoMatch
state := 0
for i := startOffset; i < len(data); i++ {
b := data[i]
for {
if _, ok := ac.gotoFn[state][b]; ok || state == 0 {
break
}
state = ac.failFn[state]
}
if next, ok := ac.gotoFn[state][b]; ok {
state = next
}
for _, patIdx := range ac.outputFn[state] {
matchStart := i - len(ac.patterns[patIdx]) + 1
matches = append(matches, ahoMatch{
PatternIndex: patIdx,
Start: matchStart,
})
}
}
return matches
}
// findOverlappingLimit is the bounded variant used by deep scans. It stops
// collecting once limit matches have been found so adversarial input cannot
// force an unbounded match slice.
func (ac *ahoCorasick) findOverlappingLimit(data []byte, startOffset, limit int) ([]ahoMatch, bool) {
if limit <= 0 {
return nil, false
}
matches := make([]ahoMatch, 0, min(limit, 256))
state := 0
for i := startOffset; i < len(data); i++ {
b := data[i]
for {
if _, ok := ac.gotoFn[state][b]; ok || state == 0 {
break
}
state = ac.failFn[state]
}
if next, ok := ac.gotoFn[state][b]; ok {
state = next
}
for _, patIdx := range ac.outputFn[state] {
matches = append(matches, ahoMatch{PatternIndex: patIdx, Start: i - len(ac.patterns[patIdx]) + 1})
if len(matches) >= limit {
return matches, true
}
}
}
return matches, false
}
+107
View File
@@ -0,0 +1,107 @@
package binsnoop
import (
"errors"
"fmt"
)
// ErrParse is returned when structure parsing fails.
var ErrParse = errors.New("structure parse error")
// StructField defines a single field in a binary structure.
type StructField struct {
Name string
Type string // "u8", "u16", "u24", "u32", "u64"
}
// typeSize returns the byte size for a type string.
func typeSize(ctype string) (int, error) {
switch ctype {
case "u8":
return 1, nil
case "u16":
return 2, nil
case "u24":
return 3, nil
case "u32":
return 4, nil
case "u64":
return 8, nil
default:
return 0, fmt.Errorf("unknown type: %s", ctype)
}
}
// StructSize returns the total byte size of a structure definition.
func StructSize(fields []StructField) int {
total := 0
for _, f := range fields {
sz, err := typeSize(f.Type)
if err != nil {
continue
}
total += sz
}
return total
}
// Parse parses a binary structure from data according to the field definitions.
// Endianness must be "big" or "little". All values are returned as int.
func Parse(data []byte, fields []StructField, endianness string) (map[string]int, error) {
structSize := StructSize(fields)
if len(data) < structSize {
return nil, ErrParse
}
result := make(map[string]int, len(fields))
offset := 0
for _, f := range fields {
sz, err := typeSize(f.Type)
if err != nil {
return nil, ErrParse
}
var value int
switch sz {
case 1:
value = int(data[offset])
case 2:
if endianness == "big" {
value = int(data[offset])<<8 | int(data[offset+1])
} else {
value = int(data[offset+1])<<8 | int(data[offset])
}
case 3:
if endianness == "big" {
value = int(data[offset])<<16 | int(data[offset+1])<<8 | int(data[offset+2])
} else {
value = int(data[offset+2])<<16 | int(data[offset+1])<<8 | int(data[offset])
}
case 4:
if endianness == "big" {
value = int(data[offset])<<24 | int(data[offset+1])<<16 | int(data[offset+2])<<8 | int(data[offset+3])
} else {
value = int(data[offset+3])<<24 | int(data[offset+2])<<16 | int(data[offset+1])<<8 | int(data[offset])
}
case 8:
if endianness == "big" {
value = int(uint64(data[offset])<<56 | uint64(data[offset+1])<<48 |
uint64(data[offset+2])<<40 | uint64(data[offset+3])<<32 |
uint64(data[offset+4])<<24 | uint64(data[offset+5])<<16 |
uint64(data[offset+6])<<8 | uint64(data[offset+7]))
} else {
value = int(uint64(data[offset+7])<<56 | uint64(data[offset+6])<<48 |
uint64(data[offset+5])<<40 | uint64(data[offset+4])<<32 |
uint64(data[offset+3])<<24 | uint64(data[offset+2])<<16 |
uint64(data[offset+1])<<8 | uint64(data[offset]))
}
}
offset += sz
result[f.Name] = value
}
return result, nil
}
+352
View File
@@ -0,0 +1,352 @@
package binsnoop
import "sort"
// Scanner is the core scanning engine. It holds compiled signature patterns
// and performs Aho-Corasick based multi-pattern matching on binary data.
type Scanner struct {
signatureCount int
patternCount int
shortSignatures []Signature
patterns [][]byte
patternSigTable map[int]Signature
extractorLookup map[string]*Extractor
ac *ahoCorasick
}
// compile builds the Aho-Corasick automaton from loaded patterns.
func (s *Scanner) compile() {
if len(s.patterns) > 0 {
s.ac = newAhoCorasick(s.patterns)
}
}
// Scan searches data for all known file signatures and returns validated results.
// This implements the same algorithm as the Rust binwalk scanner:
// 1. Check short signatures at file offset 0
// 2. Run Aho-Corasick scan with restart on high-confidence sized matches
// 3. Sort and de-conflict overlapping results
// 4. Estimate sizes for signatures with unknown size
func (s *Scanner) Scan(data []byte) []SignatureResult {
var fileMap []SignatureResult
nextValidOffset := 0
availableData := len(data)
// Phase 1: Check short signatures at offset 0
for i := range s.shortSignatures {
sig := &s.shortSignatures[i]
for _, magic := range sig.Magic {
magicStart := sig.MagicOffset
magicEnd := magicStart + len(magic)
if len(data) >= magicEnd && bytesEqual(data[magicStart:magicEnd], magic) {
result, err := sig.Parser(data, magicStart)
if err != nil {
continue
}
autoPopulate(result, sig)
fileMap = append(fileMap, *result)
if result.Confidence >= ConfidenceHigh {
nextValidOffset = result.Offset + result.Size
}
break
}
}
}
// Phase 2: Aho-Corasick scan.
//
// The automaton is run exactly once over the whole buffer; we then walk
// the matches in position order. When a high/medium-confidence sized
// signature is accepted, nextValidOffset jumps past it and any later
// matches that start inside that consumed region are skipped. This is the
// same result the binwalk scanner produces, but without re-running the
// search from every new offset — a previous implementation rescanned the
// tail of the file on each accepted match, which is O(n^2) on inputs that
// contain many closely-spaced signatures (e.g. real firmware images) and
// could appear to hang.
if s.ac == nil {
return s.postProcess(data, fileMap)
}
matches := s.ac.findOverlapping(data, nextValidOffset)
for _, m := range matches {
magicOffset := m.Start
if magicOffset < nextValidOffset {
continue
}
sig, ok := s.patternSigTable[m.PatternIndex]
if !ok {
continue
}
result, err := sig.Parser(data, magicOffset)
if err != nil {
continue
}
sigEnd := result.Offset + result.Size
if sigEnd > availableData {
continue
}
autoPopulate(result, &sig)
fileMap = append(fileMap, *result)
if result.Confidence >= ConfidenceMedium && result.Size > 0 {
nextValidOffset = sigEnd
}
}
return s.postProcess(data, fileMap)
}
// postProcess sorts results, removes conflicts, and estimates unknown sizes.
func (s *Scanner) postProcess(data []byte, fileMap []SignatureResult) []SignatureResult {
if len(fileMap) == 0 {
return fileMap
}
// Sort by offset
sort.Slice(fileMap, func(i, j int) bool {
return fileMap[i].Offset < fileMap[j].Offset
})
// De-conflict overlapping signatures in a single pass, filtering in place.
// (The previous implementation removed elements with
// append(fileMap[:i], fileMap[i+1:]...) inside the loop; each removal is
// O(n), so a file with many overlapping hits was O(n^2) and could appear
// to hang.) Removed entries are dropped; on a duplicate offset the
// higher-confidence signature wins, matching the original behaviour.
out := fileMap[:0]
nextValidOffset := 0
for _, thisSig := range fileMap {
// Duplicate offset: keep only the highest-confidence signature.
if len(out) > 0 && thisSig.Offset == out[len(out)-1].Offset {
if thisSig.Confidence > out[len(out)-1].Confidence {
out[len(out)-1] = thisSig
if thisSig.Confidence >= ConfidenceMedium {
nextValidOffset = thisSig.Offset + thisSig.Size
}
}
continue
}
// Drop signatures that fall inside a previously accepted sized region.
if thisSig.Offset < nextValidOffset {
continue
}
// Drop signatures whose declared size runs past EOF or wraps.
remainingSize := len(data) - thisSig.Offset
if thisSig.Size > remainingSize || (thisSig.Offset+thisSig.Size) < 0 {
continue
}
out = append(out, thisSig)
if thisSig.Confidence >= ConfidenceMedium {
nextValidOffset = thisSig.Offset + thisSig.Size
}
}
fileMap = out
// Estimate sizes for signatures with unknown size (0): extend each to the
// offset of the next medium-or-higher confidence signature, or to EOF.
// Computed in a single right-to-left pass to stay O(n).
nextMediumOffset := len(data)
for i := len(fileMap) - 1; i >= 0; i-- {
if fileMap[i].Size == 0 {
fileMap[i].Size = nextMediumOffset - fileMap[i].Offset
}
if fileMap[i].Confidence >= ConfidenceMedium {
nextMediumOffset = fileMap[i].Offset
}
}
return fileMap
}
// ScanDeep validates signatures without suppressing hits inside an already
// identified container. Results are bounded and annotated with containment.
func (s *Scanner) ScanDeep(data []byte, maxResults int) DeepScan {
if maxResults <= 0 || maxResults > 4096 {
maxResults = 4096
}
result := DeepScan{}
if len(data) == 0 {
return result
}
results := make([]SignatureResult, 0)
seen := make(map[struct {
offset int
name string
}]int)
add := func(parsed *SignatureResult, sig *Signature) {
if parsed == nil || parsed.Offset < 0 || parsed.Offset > len(data) {
return
}
if parsed.Size < 0 || parsed.Size > len(data)-parsed.Offset {
return
}
autoPopulate(parsed, sig)
key := struct {
offset int
name string
}{parsed.Offset, parsed.Name}
if idx, ok := seen[key]; ok {
if parsed.Confidence > results[idx].Confidence {
results[idx] = *parsed
}
return
}
if len(results) >= maxResults {
result.Truncated = true
return
}
seen[key] = len(results)
results = append(results, *parsed)
}
for i := range s.shortSignatures {
sig := &s.shortSignatures[i]
for _, magic := range sig.Magic {
start := sig.MagicOffset
end := start + len(magic)
if start >= 0 && end <= len(data) && bytesEqual(data[start:end], magic) {
parsed, err := sig.Parser(data, start)
if err == nil {
add(parsed, sig)
}
break
}
}
}
if s.ac != nil && len(results) < maxResults {
matchLimit := maxResults * 16
matches, truncated := s.ac.findOverlappingLimit(data, 0, matchLimit)
result.Truncated = result.Truncated || truncated
for _, match := range matches {
sig, ok := s.patternSigTable[match.PatternIndex]
if !ok {
continue
}
parsed, err := sig.Parser(data, match.Start)
if err != nil {
continue
}
add(parsed, &sig)
if len(results) >= maxResults {
result.Truncated = true
break
}
}
}
sort.Slice(results, func(i, j int) bool {
if results[i].Offset != results[j].Offset {
return results[i].Offset < results[j].Offset
}
if results[i].Confidence != results[j].Confidence {
return results[i].Confidence > results[j].Confidence
}
return results[i].Size > results[j].Size
})
unique := results[:0]
for _, hit := range results {
if len(unique) > 0 && unique[len(unique)-1].Offset == hit.Offset {
continue
}
unique = append(unique, hit)
}
results = unique
nextOffset := len(data)
for i := len(results) - 1; i >= 0; i-- {
if results[i].Size == 0 {
results[i].Size = nextOffset - results[i].Offset
}
nextOffset = results[i].Offset
}
active := make([]int, 0)
for _, hit := range results {
end := hit.Offset + hit.Size
kept := active[:0]
for _, idx := range active {
if result.Results[idx].End > hit.Offset {
kept = append(kept, idx)
}
}
active = kept
parent := -1
overlap := false
for i := len(active) - 1; i >= 0; i-- {
candidate := result.Results[active[i]]
if candidate.End >= end && candidate.Offset < hit.Offset {
parent = active[i]
break
}
overlap = true
}
deep := DeepScanResult{SignatureResult: hit, End: end, ParentOffset: -1, Overlap: overlap}
if parent >= 0 {
deep.ParentOffset = result.Results[parent].Offset
deep.Depth = result.Results[parent].Depth + 1
}
result.Results = append(result.Results, deep)
active = append(active, len(result.Results)-1)
}
return result
}
// Extract performs extraction on all extractable signatures in the file map.
func (s *Scanner) Extract(data []byte, filePath string, fileMap []SignatureResult) map[string]*ExtractionResult {
results := make(map[string]*ExtractionResult)
for _, sig := range fileMap {
if sig.ExtractionDeclined {
continue
}
ext, ok := s.extractorLookup[sig.Name]
if !ok || ext == nil {
continue
}
result := ExecuteExtraction(data, filePath, &sig, ext)
if !result.Success && sig.Size < len(data)-sig.Offset {
// Retry with all available data
expandedSig := sig
expandedSig.Size = len(data) - sig.Offset
result = ExecuteExtraction(data, filePath, &expandedSig, ext)
}
results[sig.ID] = result
}
return results
}
// bytesEqual compares two byte slices.
func bytesEqual(a, b []byte) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}
// autoPopulate fills in auto-generated fields on a SignatureResult.
func autoPopulate(result *SignatureResult, sig *Signature) {
result.ID = newUUID()
result.Name = sig.Name
result.AlwaysDisplay = sig.AlwaysDisplay
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+795
View File
@@ -0,0 +1,795 @@
package binsnoop
import (
"encoding/base64"
"fmt"
"strings"
)
// ---- AES descriptions ----
const (
DescriptionAESSBox = "AES S-Box"
DescriptionAESFT = "AES Forward Table"
DescriptionAESRT = "AES Reverse Table"
DescriptionAESRCON = "AES RCON"
DescriptionAESAcc = "AES Acceleration Table"
)
// aesSBoxMagic returns magic bytes for AES S-Box (forward and reverse).
func aesSBoxMagic() [][]byte {
return [][]byte{
{0x63, 0x7C, 0x77, 0x7B, 0xF2, 0x6B, 0x6F, 0xC5}, // Forward S-Box
{0x52, 0x09, 0x6A, 0xD5, 0x30, 0x36, 0xA5, 0x38}, // Reverse S-Box
}
}
// aesSBoxParser validates AES S-Box signatures.
func aesSBoxParser(data []byte, offset int) (*SignatureResult, error) {
return &SignatureResult{
Offset: offset,
Description: DescriptionAESSBox,
Confidence: ConfidenceLow,
}, nil
}
// aesForwardTableMagic returns magic bytes for AES Forward Table.
func aesForwardTableMagic() [][]byte {
return [][]byte{
{0xC6, 0x63, 0x63, 0xA5, 0xF8, 0x7C, 0x7C, 0x84, 0xEE, 0x77, 0x77, 0x99, 0xF6, 0x7B, 0x7B, 0x8D},
}
}
// aesForwardTableParser validates AES Forward Table signatures.
func aesForwardTableParser(data []byte, offset int) (*SignatureResult, error) {
return &SignatureResult{
Offset: offset,
Description: DescriptionAESFT,
Confidence: ConfidenceLow,
}, nil
}
// aesReverseTableMagic returns magic bytes for AES Reverse Table.
func aesReverseTableMagic() [][]byte {
return [][]byte{
{0x51, 0xF4, 0xA7, 0x50, 0x7E, 0x41, 0x65, 0x53, 0x1A, 0x17, 0xA4, 0xC3, 0x3A, 0x27, 0x5E, 0x96},
}
}
// aesReverseTableParser validates AES Reverse Table signatures.
func aesReverseTableParser(data []byte, offset int) (*SignatureResult, error) {
return &SignatureResult{
Offset: offset,
Description: DescriptionAESRT,
Confidence: ConfidenceLow,
}, nil
}
// aesRconMagic returns magic bytes for AES RCON.
func aesRconMagic() [][]byte {
return [][]byte{
{0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80, 0x1B, 0x36},
{0x01, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00,
0x10, 0x00, 0x00, 0x00, 0x20, 0x00, 0x00, 0x00, 0x40, 0x00, 0x00, 0x00, 0x80, 0x00, 0x00, 0x00,
0x1B, 0x00, 0x00, 0x00, 0x36, 0x00, 0x00, 0x00},
}
}
// aesRconParser validates AES RCON signatures.
func aesRconParser(data []byte, offset int) (*SignatureResult, error) {
return &SignatureResult{
Offset: offset,
Description: DescriptionAESRCON,
Confidence: ConfidenceLow,
}, nil
}
// aesAccelerationTableMagic returns magic bytes for AES Acceleration Table.
func aesAccelerationTableMagic() [][]byte {
return [][]byte{
// combined sbox x2
{0xA5, 0x84, 0x99, 0x8D, 0x0D, 0xBD, 0xB1, 0x54, 0x50, 0x03, 0xA9, 0x7D, 0x19, 0x62, 0xE6, 0x9A},
// combined sbox x3
{0xC6, 0xF8, 0xEE, 0xF6, 0xFF, 0xD6, 0xDE, 0x91, 0x60, 0x02, 0xCE, 0x56, 0xE7, 0xB5, 0x4D, 0xEC},
// Galois mult 2
{0x00, 0x02, 0x04, 0x06, 0x08, 0x0a, 0x0c, 0x0e, 0x10, 0x12, 0x14, 0x16, 0x18, 0x1a, 0x1c, 0x1e,
0x20, 0x22, 0x24, 0x26, 0x28, 0x2a, 0x2c, 0x2e},
// Galois mult 3
{0x00, 0x03, 0x06, 0x05, 0x0c, 0x0f, 0x0a, 0x09, 0x18, 0x1b, 0x1e, 0x1d, 0x14, 0x17, 0x12, 0x11,
0x30, 0x33, 0x36, 0x35, 0x3c, 0x3f, 0x3a, 0x39},
// Galois mult 9
{0x00, 0x09, 0x12, 0x1b, 0x24, 0x2d, 0x36, 0x3f, 0x48, 0x41, 0x5a, 0x53, 0x6c, 0x65, 0x7e, 0x77,
0x90, 0x99, 0x82, 0x8b, 0xb4, 0xbd, 0xa6, 0xaf},
// Galois mult 11
{0x00, 0x0b, 0x16, 0x1d, 0x2c, 0x27, 0x3a, 0x31, 0x58, 0x53, 0x4e, 0x45, 0x74, 0x7f, 0x62, 0x69,
0xb0, 0xbb, 0xa6, 0xad, 0x9c, 0x97, 0x8a, 0x81},
// Galois mult 13
{0x00, 0x0d, 0x1a, 0x17, 0x34, 0x39, 0x2e, 0x23, 0x68, 0x65, 0x72, 0x7f, 0x5c, 0x51, 0x46, 0x4b,
0xd0, 0xdd, 0xca, 0xc7, 0xe4, 0xe9, 0xfe, 0xf3},
// Galois mult 14
{0x00, 0x0e, 0x1c, 0x12, 0x38, 0x36, 0x24, 0x2a, 0x70, 0x7e, 0x6c, 0x62, 0x48, 0x46, 0x54, 0x5a,
0xe0, 0xee, 0xfc, 0xf2, 0xd8, 0xd6, 0xc4, 0xca},
}
}
// aesAccelerationTableParser validates AES Acceleration Table signatures.
func aesAccelerationTableParser(data []byte, offset int) (*SignatureResult, error) {
return &SignatureResult{
Offset: offset,
Description: DescriptionAESAcc,
Confidence: ConfidenceLow,
}, nil
}
// ---- RSA ----
const DescriptionRSA = "RSA encrypted session key"
type rsaKeyDefinition struct {
magic []byte
keySize int
usageOffset int
keyIDOffset int
validChecks map[int][][]byte
terminatorOffset int
}
func rsaKeyDefinitions() []rsaKeyDefinition {
return []rsaKeyDefinition{
// 1024b RSA key
{
magic: []byte{0x84, 0x8C, 0x03},
keySize: 1024,
keyIDOffset: 3,
terminatorOffset: 142,
usageOffset: 11,
validChecks: map[int][][]byte{
12: {
{0x04, 0x00}, {0x03, 0xff}, {0x03, 0xfe}, {0x03, 0xfd},
{0x03, 0xfc}, {0x03, 0xfb}, {0x03, 0xfa}, {0x03, 0xf9},
},
},
},
// 2048b RSA key
{
magic: []byte{0x85, 0x01, 0x0c, 0x03},
keySize: 2048,
keyIDOffset: 4,
terminatorOffset: 271,
usageOffset: 12,
validChecks: map[int][][]byte{
13: {
{0x08, 0x00}, {0x07, 0xff}, {0x07, 0xfe}, {0x07, 0xfd},
{0x07, 0xfc}, {0x07, 0xfb}, {0x07, 0xfa}, {0x07, 0xf9},
},
},
},
// 3072b RSA key
{
magic: []byte{0x85, 0x01, 0x8c, 0x03},
keySize: 3072,
keyIDOffset: 4,
terminatorOffset: 399,
usageOffset: 12,
validChecks: map[int][][]byte{
13: {
{0x0c, 0x00}, {0x0b, 0xff}, {0x0b, 0xfe}, {0x0b, 0xfd},
{0x0b, 0xfc}, {0x0b, 0xfb}, {0x0b, 0xfa}, {0x0b, 0xf9},
},
},
},
// 4096b RSA key
{
magic: []byte{0x85, 0x02, 0x0c, 0x03},
keySize: 4096,
keyIDOffset: 4,
terminatorOffset: 527,
usageOffset: 12,
validChecks: map[int][][]byte{
13: {
{0x10, 0x00}, {0x0f, 0xff}, {0x0f, 0xfe}, {0x0f, 0xfd},
{0x0f, 0xfc}, {0x0f, 0xfb}, {0x0f, 0xfa}, {0x0f, 0xf9},
},
},
},
// 8192b RSA key
{
magic: []byte{0x85, 0x04, 0x0c, 0x03},
keySize: 8192,
keyIDOffset: 4,
terminatorOffset: 1039,
usageOffset: 12,
validChecks: map[int][][]byte{
13: {
{0x20, 0x00}, {0x1f, 0xff}, {0x1f, 0xfe}, {0x1f, 0xfd},
{0x1f, 0xfc}, {0x1f, 0xfb}, {0x1f, 0xfa}, {0x1f, 0xf9},
},
},
},
}
}
// rsaMagic returns magic bytes for all known RSA key sizes.
func rsaMagic() [][]byte {
var magics [][]byte
for _, def := range rsaKeyDefinitions() {
m := make([]byte, len(def.magic))
copy(m, def.magic)
magics = append(magics, m)
}
return magics
}
type rsaKeyInfo struct {
bits int
canSign bool
canEncrypt bool
keyID int
dataSize int
}
func rsaKeyParse(rawData []byte, def *rsaKeyDefinition) (*rsaKeyInfo, error) {
const terminatorByte byte = 0xD2
const encryptOnly byte = 2
const signAndEncrypt byte = 1
const validBytesSize = 2
keyIDFields := []StructField{{"id", "u64"}}
keyDataLen := def.terminatorOffset + 1
if len(rawData) < keyDataLen {
return nil, fmt.Errorf("insufficient RSA key data")
}
keyData := rawData[:keyDataLen]
if keyData[def.terminatorOffset] != terminatorByte {
return nil, fmt.Errorf("invalid RSA terminator byte")
}
if def.keyIDOffset+8 > len(keyData) {
return nil, fmt.Errorf("key ID offset out of bounds")
}
keyIDParsed, err := Parse(keyData[def.keyIDOffset:], keyIDFields, "big")
if err != nil {
return nil, fmt.Errorf("failed to parse RSA key ID")
}
info := &rsaKeyInfo{keyID: keyIDParsed["id"]}
info.canEncrypt = keyData[def.usageOffset] == encryptOnly
info.canSign = keyData[def.usageOffset] == signAndEncrypt
if info.canSign {
info.canEncrypt = true
}
if !info.canSign && !info.canEncrypt {
return nil, fmt.Errorf("RSA key cannot sign or encrypt")
}
for checkOffset, expectedList := range def.validChecks {
if checkOffset+validBytesSize > len(keyData) {
continue
}
actual := keyData[checkOffset : checkOffset+validBytesSize]
for _, expected := range expectedList {
if len(expected) == validBytesSize && actual[0] == expected[0] && actual[1] == expected[1] {
info.bits = def.keySize
info.dataSize = keyDataLen
return info, nil
}
}
}
return nil, fmt.Errorf("RSA key validation bytes mismatch")
}
// rsaParser validates RSA encrypted session key signatures.
func rsaParser(data []byte, offset int) (*SignatureResult, error) {
result := &SignatureResult{
Offset: offset,
Description: DescriptionRSA,
Confidence: ConfidenceMedium,
}
for _, def := range rsaKeyDefinitions() {
magicEnd := offset + len(def.magic)
if magicEnd > len(data) {
continue
}
match := true
for i, b := range def.magic {
if data[offset+i] != b {
match = false
break
}
}
if !match {
continue
}
info, err := rsaKeyParse(data[offset:], &def)
if err != nil {
break
}
result.Size = info.dataSize
result.Description = fmt.Sprintf("%s, %d bits, can sign: %t, can encrypt: %t, key ID: %#018X",
result.Description, info.bits, info.canSign, info.canEncrypt, info.keyID)
return result, nil
}
return nil, fmt.Errorf("invalid RSA signature")
}
// ---- OpenSSL ----
const DescriptionOpenSSL = "OpenSSL encryption"
// opensslMagic returns magic bytes for OpenSSL encrypted files.
func opensslMagic() [][]byte {
return [][]byte{
[]byte("Salted__"),
}
}
func isSaltInvalid(salt int) bool {
const saltLen = 8
badByteCount := 0
for i := 0; i < saltLen; i++ {
saltByte := byte((salt >> (8 * i)) & 0xFF)
if saltByte == 0 || IsPrintableASCII(saltByte) {
badByteCount++
}
}
return badByteCount == saltLen
}
// opensslParser validates OpenSSL encryption signatures.
func opensslParser(data []byte, offset int) (*SignatureResult, error) {
sslFields := []StructField{{"magic", "u64"}, {"salt", "u64"}}
result := &SignatureResult{
Offset: offset,
Description: DescriptionOpenSSL,
Confidence: ConfidenceLow,
}
if offset+StructSize(sslFields) > len(data) {
return nil, fmt.Errorf("insufficient OpenSSL data")
}
sslHeader, err := Parse(data[offset:], sslFields, "big")
if err != nil {
return nil, fmt.Errorf("failed to parse OpenSSL header")
}
salt := sslHeader["salt"]
if salt == 0 {
return nil, fmt.Errorf("OpenSSL salt is zero")
}
if isSaltInvalid(salt) {
return nil, fmt.Errorf("OpenSSL salt is invalid")
}
if offset == 0 {
result.Confidence = ConfidenceMedium
}
result.Description = fmt.Sprintf("%s, salt: %#X", result.Description, salt)
return result, nil
}
// ---- GPG ----
const DescriptionGPGSigned = "GPG signed file"
// gpgSignedMagic returns magic bytes for GPG signed files.
func gpgSignedMagic() [][]byte {
return [][]byte{
{0xA3, 0x01},
}
}
// gpgSignedParser validates GPG signed file signatures.
// The Rust version does a zlib decompression dry run; here we do a simplified validation.
func gpgSignedParser(data []byte, offset int) (*SignatureResult, error) {
result := &SignatureResult{
Offset: offset,
Confidence: ConfidenceHigh,
Description: DescriptionGPGSigned,
}
// GPG signed files are zlib compressed with magic bytes replaced.
// Validate there is enough data and the zlib-like stream follows.
if offset+4 > len(data) {
return nil, fmt.Errorf("insufficient GPG data")
}
// The third byte should be a valid zlib CMF byte (deflate method = 8)
cmf := data[offset+2]
if cmf&0x0F != 8 {
return nil, fmt.Errorf("GPG: invalid deflate method in underlying zlib stream")
}
result.Description = fmt.Sprintf("%s, compressed data at offset %d", result.Description, offset)
return result, nil
}
// ---- PEM ----
const (
DescriptionPEMCertificate = "PEM certificate"
DescriptionPEMPublicKey = "PEM public key"
DescriptionPEMPrivateKey = "PEM private key"
)
// pemCertificateMagic returns magic bytes for PEM certificates.
func pemCertificateMagic() [][]byte {
return [][]byte{
[]byte("-----BEGIN CERTIFICATE-----"),
}
}
// pemPublicKeyMagic returns magic bytes for PEM public keys.
func pemPublicKeyMagic() [][]byte {
return [][]byte{
[]byte("-----BEGIN PUBLIC KEY-----"),
[]byte("-----BEGIN RSA PUBLIC KEY-----"),
[]byte("-----BEGIN DSA PUBLIC KEY-----"),
[]byte("-----BEGIN ECDSA PUBLIC KEY-----"),
}
}
// pemPrivateKeyMagic returns magic bytes for PEM private keys.
func pemPrivateKeyMagic() [][]byte {
return [][]byte{
[]byte("-----BEGIN PRIVATE KEY-----"),
[]byte("-----BEGIN EC PRIVATE KEY-----"),
[]byte("-----BEGIN RSA PRIVATE KEY-----"),
[]byte("-----BEGIN DSA PRIVATE KEY-----"),
[]byte("-----BEGIN OPENSSH PRIVATE KEY-----"),
[]byte("-----BEGIN ANY PRIVATE KEY-----"),
[]byte("-----BEGIN ENCRYPTED PRIVATE KEY-----"),
[]byte("-----BEGIN TSS2 PRIVATE KEY-----"),
}
}
func decodePEMData(pemData []byte) (int, error) {
const delim = "--"
pemStr := string(pemData)
delimCount := 0
var base64Str strings.Builder
for _, line := range strings.Split(pemStr, "\n") {
line = strings.TrimRight(line, "\r")
if strings.HasPrefix(line, delim) {
delimCount++
if delimCount == 2 {
break
}
continue
}
base64Str.WriteString(line)
}
if base64Str.Len() == 0 {
return 0, fmt.Errorf("no PEM base64 data found")
}
decoded, err := base64.StdEncoding.DecodeString(base64Str.String())
if err != nil {
return 0, fmt.Errorf("PEM base64 decode failed: %v", err)
}
return len(decoded), nil
}
// pemFindEnd locates the end delimiter and returns the total PEM size.
func pemFindEnd(data []byte, offset int) (int, bool) {
const endPrefix = "-----END "
pemData := string(data[offset:])
idx := strings.Index(pemData, endPrefix)
if idx < 0 {
return 0, false
}
// Find end of the delimiter line
rest := pemData[idx:]
nlIdx := strings.Index(rest, "\n")
if nlIdx < 0 {
return idx + len(rest), true
}
return idx + nlIdx + 1, true
}
// pemParser validates PEM certificate and key signatures.
func pemParser(data []byte, offset int) (*SignatureResult, error) {
const minPEMLen = 26
result := &SignatureResult{
Offset: offset,
Confidence: ConfidenceHigh,
}
if offset+minPEMLen > len(data) {
return nil, fmt.Errorf("insufficient PEM data")
}
pemMagic := data[offset : offset+minPEMLen]
pemMagicStr := string(pemMagic)
// Determine type based on first 26 bytes
matched := false
for _, m := range pemPublicKeyMagic() {
if len(m) >= minPEMLen && strings.HasPrefix(string(m), pemMagicStr[:minPEMLen]) {
result.Description = DescriptionPEMPublicKey
matched = true
break
}
}
if !matched {
for _, m := range pemPrivateKeyMagic() {
if len(m) >= minPEMLen && strings.HasPrefix(string(m), pemMagicStr[:minPEMLen]) {
result.Description = DescriptionPEMPrivateKey
matched = true
break
}
}
}
if !matched {
for _, m := range pemCertificateMagic() {
if len(m) >= minPEMLen && strings.HasPrefix(string(m), pemMagicStr[:minPEMLen]) {
result.Description = DescriptionPEMCertificate
matched = true
break
}
}
}
if !matched {
return nil, fmt.Errorf("PEM magic not recognised")
}
// Find the end of PEM data
pemSize, found := pemFindEnd(data, offset)
if !found || pemSize == 0 {
return nil, fmt.Errorf("PEM end delimiter not found")
}
// Validate base64 content
if offset+pemSize > len(data) {
pemSize = len(data) - offset
}
if _, err := decodePEMData(data[offset : offset+pemSize]); err != nil {
return nil, fmt.Errorf("PEM validation failed: %v", err)
}
if offset == 0 && pemSize == len(data) {
result.ExtractionDeclined = true
}
result.Size = pemSize
return result, nil
}
// ---- PKCS DER ----
const DescriptionDERHash = "PKCS DER hash"
func derHashLookups() map[string][]byte {
return map[string][]byte{
"MD5": {0x30, 0x20, 0x30, 0x0c, 0x06, 0x08, 0x2a, 0x86, 0x48, 0x86, 0xf7, 0x0d, 0x02, 0x05, 0x05, 0x00, 0x04, 0x10},
"SHA1": {0x30, 0x21, 0x30, 0x09, 0x06, 0x05, 0x2b, 0x0e, 0x03, 0x02, 0x1a, 0x05, 0x00, 0x04, 0x14},
"SHA256": {0x30, 0x31, 0x30, 0x0d, 0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x01, 0x05, 0x00, 0x04, 0x20},
"SHA384": {0x30, 0x41, 0x30, 0x0d, 0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x02, 0x05, 0x00, 0x04, 0x30},
"SHA512": {0x30, 0x51, 0x30, 0x0d, 0x06, 0x09, 0x60, 0x86, 0x48, 0x01, 0x65, 0x03, 0x04, 0x02, 0x03, 0x05, 0x00, 0x04},
}
}
// derHashMagic returns magic bytes for all DER hash types.
func derHashMagic() [][]byte {
var magics [][]byte
for _, m := range derHashLookups() {
mc := make([]byte, len(m))
copy(mc, m)
magics = append(magics, mc)
}
return magics
}
// derHashParser validates PKCS DER hash signatures.
func derHashParser(data []byte, offset int) (*SignatureResult, error) {
result := &SignatureResult{
Offset: offset,
Description: DescriptionDERHash,
Confidence: ConfidenceMedium,
}
for name, magic := range derHashLookups() {
hashEnd := offset + len(magic)
if hashEnd > len(data) {
continue
}
match := true
for i, b := range magic {
if data[offset+i] != b {
match = false
break
}
}
if match {
result.Size = len(magic)
result.Description = fmt.Sprintf("%s, %s", result.Description, name)
return result, nil
}
}
return nil, fmt.Errorf("DER hash validation failed")
}
// ---- LUKS ----
const DescriptionLUKS = "LUKS header"
// luksMagic returns magic bytes for LUKS headers.
func luksMagic() [][]byte {
return [][]byte{
{0x4C, 0x55, 0x4B, 0x53, 0xBA, 0xBE}, // "LUKS\xBA\xBE"
}
}
// luksParser validates LUKS header signatures.
func luksParser(data []byte, offset int) (*SignatureResult, error) {
const cipherAlgoStart = 8
const cipherAlgoEnd = 40
const cipherModeStart = 40
const cipherModeEnd = 72
const hashFnStart = 72
const hashFnEnd = 104
const luks2MinHeaderSize = 4032
luksBaseFields := []StructField{
{"magic_1", "u32"},
{"magic_2", "u16"},
{"version", "u16"},
{"header_size", "u64"},
}
result := &SignatureResult{
Offset: offset,
Name: "luks",
Description: DescriptionLUKS,
Confidence: ConfidenceMedium,
}
if offset+StructSize(luksBaseFields) > len(data) {
return nil, fmt.Errorf("insufficient LUKS data")
}
luksBase, err := Parse(data[offset:], luksBaseFields, "big")
if err != nil {
return nil, fmt.Errorf("failed to parse LUKS header")
}
version := luksBase["version"]
// Hash function string
if offset+hashFnEnd > len(data) {
return nil, fmt.Errorf("LUKS hash function data out of bounds")
}
hashFn := GetCString(data[offset+hashFnStart : offset+hashFnEnd])
if hashFn == "" {
return nil, fmt.Errorf("LUKS hash function string empty")
}
if version == 1 {
if offset+cipherAlgoEnd > len(data) {
return nil, fmt.Errorf("LUKS cipher data out of bounds")
}
cipherAlgo := GetCString(data[offset+cipherAlgoStart : offset+cipherAlgoEnd])
cipherMode := GetCString(data[offset+cipherModeStart : offset+cipherModeEnd])
if cipherAlgo == "" || cipherMode == "" {
return nil, fmt.Errorf("LUKS v1 cipher strings empty")
}
result.Description = fmt.Sprintf("%s, version: %d, cipher algorithm: %s, cipher mode: %s, hash fn: %s",
result.Description, version, cipherAlgo, cipherMode, hashFn)
return result, nil
} else if version == 2 {
headerSize := luksBase["header_size"]
availableData := len(data) - offset
if headerSize <= luks2MinHeaderSize || headerSize >= availableData {
return nil, fmt.Errorf("LUKS v2 header size invalid")
}
result.Description = fmt.Sprintf("%s, version: %d, header size: %d bytes, hash fn: %s",
result.Description, version, headerSize, hashFn)
return result, nil
}
return nil, fmt.Errorf("unsupported LUKS version: %d", version)
}
// ---- Hash constants ----
const hashMagicLen = 16
const (
DescriptionCRC32 = "CRC32 polynomial table"
DescriptionSHA256 = "SHA256 hash constants"
DescriptionMD5 = "MD5 hash constants"
)
// crc32Magic returns magic bytes for CRC32 polynomial tables.
// Order matters: index 0 is big endian, index 1 is little endian.
func crc32ConstMagic() [][]byte {
return [][]byte{
// Big endian
{0x00, 0x00, 0x00, 0x00, 0x77, 0x07, 0x30, 0x96, 0xEE, 0x0E, 0x61, 0x2C, 0x99, 0x09, 0x51, 0xBA},
// Little endian
{0x00, 0x00, 0x00, 0x00, 0x96, 0x30, 0x07, 0x77, 0x2C, 0x61, 0x0E, 0xEE, 0xBA, 0x51, 0x09, 0x99},
}
}
// sha256Magic returns magic bytes for SHA256 hash constants.
func sha256ConstMagic() [][]byte {
return [][]byte{
// Big endian
{0x42, 0x8a, 0x2f, 0x98, 0x71, 0x37, 0x44, 0x91, 0xb5, 0xc0, 0xfb, 0xcf, 0xe9, 0xb5, 0xdb, 0xa5},
// Little endian
{0x98, 0x2f, 0x8a, 0x42, 0x91, 0x44, 0x37, 0x71, 0xcf, 0xfb, 0xc0, 0xb5, 0xa5, 0xdb, 0xb5, 0xe9},
}
}
// md5Magic returns magic bytes for MD5 hash constants.
func md5ConstMagic() [][]byte {
return [][]byte{
// Big endian
{0xd7, 0x6a, 0xa4, 0x78, 0xe8, 0xc7, 0xb7, 0x56, 0x24, 0x20, 0x70, 0xdb, 0xc1, 0xbd, 0xce, 0xee},
// Little endian
{0x78, 0xa4, 0x6a, 0xd7, 0x56, 0xb7, 0xc7, 0xe8, 0xdb, 0x70, 0x20, 0x24, 0xee, 0xce, 0xbd, 0xc1},
}
}
func hashEndianness(data []byte, offset int, magics [][]byte) string {
if offset+hashMagicLen > len(data) {
return "little"
}
thisMagic := data[offset : offset+hashMagicLen]
if len(magics) > 0 && len(magics[0]) == hashMagicLen {
match := true
for i := 0; i < hashMagicLen; i++ {
if thisMagic[i] != magics[0][i] {
match = false
break
}
}
if match {
return "big"
}
}
return "little"
}
// crc32Parser validates CRC32 polynomial table signatures.
func crc32Parser(data []byte, offset int) (*SignatureResult, error) {
return &SignatureResult{
Offset: offset,
Size: hashMagicLen,
Description: fmt.Sprintf("%s, %s endian", DescriptionCRC32, hashEndianness(data, offset, crc32ConstMagic())),
}, nil
}
// sha256Parser validates SHA256 hash constant signatures.
func sha256Parser(data []byte, offset int) (*SignatureResult, error) {
return &SignatureResult{
Offset: offset,
Size: hashMagicLen,
Description: fmt.Sprintf("%s, %s endian", DescriptionSHA256, hashEndianness(data, offset, sha256ConstMagic())),
}, nil
}
// md5Parser validates MD5 hash constant signatures.
func md5Parser(data []byte, offset int) (*SignatureResult, error) {
return &SignatureResult{
Offset: offset,
Size: hashMagicLen,
Description: fmt.Sprintf("%s, %s endian", DescriptionMD5, hashEndianness(data, offset, md5ConstMagic())),
}, nil
}
@@ -0,0 +1,485 @@
package binsnoop
import (
"fmt"
)
// ---------------------------------------------------------------------------
// ELF
// ---------------------------------------------------------------------------
const ELFDescription = "ELF binary"
func ELFMagic() [][]byte {
return [][]byte{
{0x7F, 0x45, 0x4C, 0x46}, // "\x7FELF"
}
}
func ELFParser(data []byte, offset int) (*SignatureResult, error) {
const elfIdentSize = 16
const elfInfoStructSize = 8
const expectedVersion = 1
elfIdentFields := []StructField{
{"magic", "u32"},
{"class", "u8"},
{"endianness", "u8"},
{"version", "u8"},
{"osabi", "u8"},
{"abiversion", "u8"},
{"padding_1", "u32"},
{"padding_2", "u24"},
}
elfInfoFields := []StructField{
{"type", "u16"},
{"machine", "u16"},
{"version", "u32"},
}
elfClasses := map[int]int{
1: 32,
2: 64,
}
elfEndianness := map[int]string{
1: "little",
2: "big",
}
elfOSABI := map[int]string{
0: "System-V (Unix)",
1: "HP-UX",
2: "NetBSD",
3: "Linux",
4: "GNU Hurd",
5: "86Open",
6: "Solaris",
7: "AIX",
8: "IRIX",
9: "FreeBSD",
10: "Tru64",
11: "Novell Modesto",
12: "OpenBSD",
13: "OpenVMS",
14: "NonStop Kernel",
15: "AROS",
16: "FenixOS",
17: "Nuxi CloudABI",
18: "OpenVOS",
97: "ARM ABI",
102: "Cell LV2",
202: "Cafe OS",
255: "embedded",
}
elfTypes := map[int]string{
0: "no file type",
1: "relocatable",
2: "executable",
3: "shared object",
4: "core file",
}
elfMachines := map[int]string{
0: "no machine",
1: "AT&T WE 32100",
2: "SPARC",
3: "x86",
4: "Motorola 68k",
5: "Motorola 88k",
6: "Intel MCU",
7: "Intel 80860",
8: "MIPS",
9: "IBM System/370",
10: "MIPS RS3000",
11: "RS6000",
15: "HP PA-RISC",
16: "nCUBE",
17: "Fujitsu VPP500",
18: "SPARC32PLUS",
19: "Intel 80960",
20: "PowerPC",
21: "PowerPC 64-bit",
22: "S390",
23: "IBM SPU/SPC",
24: "cisco SVIP",
25: "cisco 7200",
36: "NEC V800",
37: "Fujitsu FR20",
38: "TRW RH-32",
39: "Motorola RCE",
40: "ARM",
41: "Digital Alpha",
42: "SuperH",
43: "SPARCv9",
44: "Siemens TriCore embedded processor",
45: "Argonaut RISC Core",
46: "Hitachi H8/300",
47: "Hitachi H8/300H",
48: "Hitachi H8S",
49: "Hitachi H8/500",
50: "IA-64",
51: "Stanford MIPS-X",
52: "Motorola ColdFire",
53: "Motorola M68HC12",
54: "Fujitsu MMA Multimedia Accelerator",
55: "Siemens PCP",
56: "Sony nCPU embedded RISC processor",
57: "Denso NDR1 microprocessor",
58: "Motorola StarCore",
59: "Toyota ME16",
60: "STMicroelectronics ST100",
61: "Advanced Logic TinyJ embedded processor",
62: "AMD X86-64",
63: "Sony DSP processor",
64: "PDP-10",
65: "PDP-11",
66: "Siemens FX66",
67: "STMicroelectronics ST9+",
68: "STMicroelectronics ST7",
69: "Motorola MC68HC16",
70: "Motorola MC68HC11",
71: "Motorola MC68HC08",
72: "Motorola MC68HC05",
73: "Silicon Graphics SVx",
74: "STMicroelectonrics ST19",
75: "Digital VAX",
76: "Axis Communications 32-bit CPU",
77: "Infineon Technologies 32-bit CPU",
78: "Element 14 64-bit DSP",
79: "LSI Logic 16-bit DSP",
80: "MMIX",
81: "Harvard machine-independent",
82: "SiTera Prism",
83: "Atmel AVR 8-bit",
84: "Fujitsu FR30",
85: "Mitsubishi D10V",
86: "Mitsubishi D30V",
87: "NEC v850",
88: "Renesas M32R",
89: "Matsushita MN10300",
90: "Matsushita MN10200",
91: "picoJava",
92: "OpenRISC",
93: "Synopsys ARCompact ARC700 cores",
94: "Tensilica Xtensa",
95: "Alphamosaic VideoCore",
96: "Thompson Multimedia",
97: "NatSemi 32k",
98: "Tenor Network TPC",
99: "Trebia SNP 1000",
100: "STMicroelectronics ST200",
101: "Ubicom IP2022",
102: "MAX Processor",
103: "NatSemi CompactRISC",
104: "Fujitsu F2MC16",
105: "TI msp430",
106: "Analog Devices Blackfin",
107: "S1C33 Family of Seiko Epson",
108: "Sharp embedded",
109: "Arca RISC",
110: "PKU-Unity Ltd.",
111: "eXcess: 16/32/64-bit",
112: "Icera Deep Execution Processor",
113: "Altera Nios II",
114: "NatSemi CRX",
115: "Motorola XGATE",
116: "Infineon C16x/XC16x",
117: "Renesas M16C series",
118: "Microchip dsPIC30F",
119: "Freescale RISC core",
120: "Renesas M32C series",
131: "Altium TSK3000 core",
132: "Freescale RS08",
134: "Cyan Technology eCOG2",
135: "Sunplus S+core7 RISC",
136: "New Japan Radio (NJR) 24-bit DSP",
137: "Broadcom VideoCore III processor",
138: "LatticeMico32",
139: "Seiko Epson C17 family",
140: "TMS320C6000",
141: "TMS320C2000",
142: "TMS320C55x",
144: "TI Programmable Realtime Unit",
160: "STMicroelectronics 64bit VLIW DSP",
161: "Cypress M8C",
162: "Renesas R32C series",
163: "NXP TriMedia family",
164: "Qualcomm DSP6",
165: "Intel 8051 and variants",
166: "STMicroelectronics STxP7x family",
167: "Andes embedded RISC",
168: "Cyan eCOG1X family",
169: "Dallas MAXQ30",
170: "New Japan Radio (NJR) 16-bit DSP",
171: "M2000 Reconfigurable RISC",
172: "Cray NV2 vector architecture",
173: "Renesas RX family",
174: "META",
175: "MCST Elbrus e2k",
176: "Cyan Technology eCOG16 family",
177: "NatSemi CompactRISC",
178: "Freescale Extended Time Processing Unit",
179: "Infineon SLE9X",
180: "Intel L1OM",
181: "Intel K1OM",
183: "ARM 64-bit",
185: "Atmel 32-bit family",
186: "STMicroeletronics STM8 8-bit",
187: "Tilera TILE64",
188: "Tilera TILEPro",
189: "Xilinx MicroBlaze 32-bit RISC",
190: "NVIDIA CUDA architecture",
191: "Tilera TILE-Gx",
195: "Synopsys ARCv2/HS3x/HS4x cores",
197: "Renesas RL78 family",
199: "Renesas 78K0R",
200: "Freescale 56800EX",
201: "Beyond BA1",
202: "Beyond BA2",
203: "XMOS xCORE",
204: "Microchip 8-bit PIC(r)",
210: "KM211 KM32",
211: "KM211 KMX32",
212: "KM211 KMX16",
213: "KM211 KMX8",
214: "KM211 KVARC",
215: "Paneve CDP",
216: "Cognitive Smart Memory",
217: "iCelero CoolEngine",
218: "Nanoradio Optimized RISC",
219: "CSR Kalimba architecture family",
220: "Zilog Z80",
221: "Controls and Data Services VISIUMcore processor",
222: "FTDI Chip FT32 high performance 32-bit RISC architecture",
223: "Moxie processor family",
224: "AMD GPU architecture",
243: "RISC-V",
244: "Lanai 32-bit processor",
245: "CEVA Processor Architecture Family",
246: "CEVA X2 Processor Family",
247: "Berkeley Packet Filter",
248: "Graphcore Intelligent Processing Unit",
249: "Imagination Technologies",
250: "Netronome Flow Processor",
251: "NEC Vector Engine",
252: "C-SKY processor family",
253: "Synopsys ARCv3 64-bit ISA/HS6x cores",
254: "MOS Technology MCS 6502 processor",
255: "Synopsys ARCv3 32-bit",
256: "Kalray VLIW core of the MPPA family",
257: "WDC 65C816",
258: "LoongArch",
259: "ChipON KungFu32",
}
elfData := data[offset:]
if len(elfData) < elfIdentSize+elfInfoStructSize {
return nil, fmt.Errorf("ELF: not enough data")
}
// Parse the e_ident (endianness doesn't matter for single-byte fields)
eIdent, err := Parse(elfData, elfIdentFields, "little")
if err != nil {
return nil, fmt.Errorf("ELF: e_ident parse error: %w", err)
}
// Sanity check padding fields
if eIdent["padding_1"] != 0 || eIdent["padding_2"] != 0 {
return nil, fmt.Errorf("ELF: non-zero padding in e_ident")
}
if eIdent["version"] != expectedVersion {
return nil, fmt.Errorf("ELF: unexpected e_ident version %d", eIdent["version"])
}
classVal, classOK := elfClasses[eIdent["class"]]
if !classOK {
return nil, fmt.Errorf("ELF: unknown class %d", eIdent["class"])
}
endianStr, endianOK := elfEndianness[eIdent["endianness"]]
if !endianOK {
return nil, fmt.Errorf("ELF: unknown endianness %d", eIdent["endianness"])
}
osabiStr, osabiOK := elfOSABI[eIdent["osabi"]]
if !osabiOK {
return nil, fmt.Errorf("ELF: unknown OS/ABI %d", eIdent["osabi"])
}
// Parse the rest of the ELF header with the correct endianness
elfInfoData := elfData[elfIdentSize : elfIdentSize+elfInfoStructSize]
elfInfo, err := Parse(elfInfoData, elfInfoFields, endianStr)
if err != nil {
return nil, fmt.Errorf("ELF: info parse error: %w", err)
}
if elfInfo["version"] != expectedVersion {
return nil, fmt.Errorf("ELF: unexpected info version %d", elfInfo["version"])
}
typeStr, typeOK := elfTypes[elfInfo["type"]]
if !typeOK {
return nil, fmt.Errorf("ELF: unknown type %d", elfInfo["type"])
}
machineStr := "Unknown"
if m, ok := elfMachines[elfInfo["machine"]]; ok {
machineStr = m
}
return &SignatureResult{
Offset: offset,
Name: "elf",
Description: fmt.Sprintf("%s, %d-bit %s, %s for %s, %s endian", ELFDescription, classVal, typeStr, machineStr, osabiStr, endianStr),
Confidence: ConfidenceMedium,
}, nil
}
// ---------------------------------------------------------------------------
// PE
// ---------------------------------------------------------------------------
const PEDescription = "Windows PE binary"
func PEMagic() [][]byte {
// First 16 bytes of common DOS headers (e_magic through e_ss)
return [][]byte{
{0x4D, 0x5A, 0x90, 0x00, 0x03, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0xFF, 0xFF, 0x00, 0x00},
{0x4D, 0x5A, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00},
}
}
func PEParser(data []byte, offset int) (*SignatureResult, error) {
const peMagicValue = 0x00004550 // "PE\x00\x00"
dosFields := []StructField{
{"e_magic", "u16"},
{"e_cblp", "u16"},
{"e_cp", "u16"},
{"e_crlc", "u16"},
{"e_cparhdr", "u16"},
{"e_minalloc", "u16"},
{"e_maxalloc", "u16"},
{"e_ss", "u16"},
{"e_sp", "u16"},
{"e_csum", "u16"},
{"e_ip", "u16"},
{"e_cs", "u16"},
{"e_lfarlc", "u16"},
{"e_ovno", "u16"},
{"e_res_1", "u16"},
{"e_res_2", "u16"},
{"e_res_3", "u16"},
{"e_res_4", "u16"},
{"e_oemid", "u16"},
{"e_oeminfo", "u16"},
{"e_res_5", "u16"},
{"e_res_6", "u16"},
{"e_res_7", "u16"},
{"e_res_8", "u16"},
{"e_res_9", "u16"},
{"e_res_10", "u16"},
{"e_res_11", "u16"},
{"e_res_12", "u16"},
{"e_res_13", "u16"},
{"e_res_14", "u16"},
{"e_lfanew", "u32"},
}
peFields := []StructField{
{"magic", "u32"},
{"machine", "u16"},
{"number_of_sections", "u16"},
{"timestamp", "u32"},
{"symbol_table_ptr", "u32"},
{"number_of_symbols", "u32"},
{"optional_header_size", "u16"},
{"characteristics", "u16"},
}
knownMachineTypes := map[int]string{
0x0: "Unknown",
0x184: "Alpha32",
0x284: "Alpha64",
0x1D3: "Matsushita AM33",
0x8664: "Intel x86-64",
0x1C0: "ARM",
0xAA64: "ARM-64",
0x1C4: "ARM Thumb2",
0xEBC: "EFI",
0x14C: "Intel x86",
0x200: "Intel Itanium",
0x6232: "LoongArch 32-bit",
0x6264: "LoongArch 64-bit",
0x9041: "Mitsubishi M32R",
0x266: "MIPS16",
0x366: "MIPS with FPU",
0x466: "MIPS16 with FPU",
0x1F0: "PowerPC",
0x1F1: "PowerPC with FPU",
0x5032: "RISC-V 32-bit",
0x5064: "RISC-V 64-bit",
0x5128: "RISC-V 128-bit",
0x1A2: "Hitachi SH3",
0x1A3: "Hitachi SH3 DSP",
0x1A6: "Hitachi SH4",
0x1A8: "Hitachi SH5",
0x1C2: "Thumb",
0x169: "MIPS WCEv2",
}
peData := data[offset:]
dosStructSize := StructSize(dosFields)
peStructSize := StructSize(peFields)
if len(peData) < dosStructSize {
return nil, fmt.Errorf("PE: not enough data for DOS header")
}
dosHeader, err := Parse(peData, dosFields, "little")
if err != nil {
return nil, fmt.Errorf("PE: DOS header parse error: %w", err)
}
// Sanity check the reserved header fields; they should all be 0
for _, field := range []string{
"e_res_1", "e_res_2", "e_res_3", "e_res_4",
"e_res_5", "e_res_6", "e_res_7", "e_res_8",
"e_res_9", "e_res_10", "e_res_11", "e_res_12",
"e_res_13", "e_res_14",
} {
if dosHeader[field] != 0 {
return nil, fmt.Errorf("PE: reserved field %s is non-zero", field)
}
}
peHeaderStart := dosHeader["e_lfanew"]
peHeaderEnd := peHeaderStart + peStructSize
if peHeaderEnd > len(peData) {
return nil, fmt.Errorf("PE: e_lfanew points beyond available data")
}
peHeader, err := Parse(peData[peHeaderStart:peHeaderEnd], peFields, "little")
if err != nil {
return nil, fmt.Errorf("PE: PE header parse error: %w", err)
}
if peHeader["magic"] != peMagicValue {
return nil, fmt.Errorf("PE: invalid PE magic 0x%08X", peHeader["magic"])
}
machineStr, machineOK := knownMachineTypes[peHeader["machine"]]
if !machineOK {
return nil, fmt.Errorf("PE: unknown machine type 0x%X", peHeader["machine"])
}
return &SignatureResult{
Offset: offset,
Description: fmt.Sprintf("%s, machine type: %s", PEDescription, machineStr),
Confidence: ConfidenceMedium,
}, nil
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+677
View File
@@ -0,0 +1,677 @@
package binsnoop
import (
"bytes"
"fmt"
"strings"
)
// ---------------------------------------------------------------------------
// PNG
// ---------------------------------------------------------------------------
const PNGDescription = "PNG image"
func PNGMagic() [][]byte {
// PNG magic followed by IHDR chunk size (0x0000000D) and IHDR chunk type
return [][]byte{
{0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A, 0x00, 0x00, 0x00, 0x0D, 0x49, 0x48, 0x44, 0x52},
}
}
func PNGParser(data []byte, offset int) (*SignatureResult, error) {
const pngHeaderLen = 8
const crcSize = 4
const iendChunkType = 0x49454E44
chunkFields := []StructField{
{"length", "u32"},
{"type", "u32"},
}
chunkHeaderSize := StructSize(chunkFields)
if offset+pngHeaderLen >= len(data) {
return nil, fmt.Errorf("PNG: not enough data for header")
}
// Walk chunks starting after the 8-byte PNG header
chunkData := data[offset+pngHeaderLen:]
available := len(chunkData)
chunkOffset := 0
var previousOffset *int
for IsOffsetSafe(available, chunkOffset, previousOffset) {
if chunkOffset+chunkHeaderSize > available {
break
}
hdr, err := Parse(chunkData[chunkOffset:], chunkFields, "big")
if err != nil {
break
}
isLast := hdr["type"] == iendChunkType
totalChunkSize := chunkHeaderSize + hdr["length"] + crcSize
prev := chunkOffset
previousOffset = &prev
chunkOffset += totalChunkSize
if isLast {
pngSize := pngHeaderLen + chunkOffset
return &SignatureResult{
Offset: offset,
Description: fmt.Sprintf("%s, total size: %d bytes", PNGDescription, pngSize),
Confidence: ConfidenceHigh,
Size: pngSize,
}, nil
}
}
return nil, fmt.Errorf("PNG: failed to walk chunks to IEND")
}
// ---------------------------------------------------------------------------
// JPEG
// ---------------------------------------------------------------------------
const JPEGDescription = "JPEG image"
func JPEGMagic() [][]byte {
return [][]byte{
{0xFF, 0xD8, 0xFF, 0xE0, 0x00, 0x10, 0x4A, 0x46, 0x49, 0x46, 0x00}, // JFIF
{0xFF, 0xD8, 0xFF, 0xE1}, // EXIF
{0xFF, 0xD8, 0xFF, 0xDB}, // DQT
}
}
func JPEGParser(data []byte, offset int) (*SignatureResult, error) {
const markerMagic byte = 0xFF
const sosMarker byte = 0xDA
const eofMarker byte = 0xD9
noLengthMarkers := []byte{
0x00, 0x01, 0xD0, 0xD1, 0xD2, 0xD3, 0xD4, 0xD5, 0xD6, 0xD7, 0xD8, eofMarker,
}
sosSkipMarkers := []byte{
0x00, 0xD0, 0xD1, 0xD2, 0xD3, 0xD4, 0xD5, 0xD6, 0xD7,
}
jpegData := data[offset:]
nextMarkerOffset := 0
for {
if nextMarkerOffset >= len(jpegData) {
break
}
magic := jpegData[nextMarkerOffset]
if magic != markerMagic {
break
}
nextMarkerOffset++
if nextMarkerOffset >= len(jpegData) {
break
}
markerID := jpegData[nextMarkerOffset]
nextMarkerOffset++
// If this marker has a length field
if !byteInSlice(markerID, noLengthMarkers) {
if nextMarkerOffset+2 > len(jpegData) {
break
}
segLen := int(jpegData[nextMarkerOffset])<<8 | int(jpegData[nextMarkerOffset+1])
nextMarkerOffset += segLen
}
// SOS marker: scan until next valid marker
if markerID == sosMarker {
for {
if nextMarkerOffset+2 > len(jpegData) {
break
}
if jpegData[nextMarkerOffset] == markerMagic && !byteInSlice(jpegData[nextMarkerOffset+1], sosSkipMarkers) {
break
}
nextMarkerOffset++
}
}
// EOF marker
if markerID == eofMarker {
jpegSize := nextMarkerOffset
return &SignatureResult{
Offset: offset,
Description: fmt.Sprintf("%s, total size: %d bytes", JPEGDescription, jpegSize),
Confidence: ConfidenceMedium,
Size: jpegSize,
}, nil
}
}
return nil, fmt.Errorf("JPEG: failed to find EOI marker")
}
// ---------------------------------------------------------------------------
// BMP
// ---------------------------------------------------------------------------
const BMPDescription = "BMP image (Bitmap)"
func BMPMagic() [][]byte {
return [][]byte{
{0x42, 0x4D}, // "BM"
}
}
func BMPParser(data []byte, offset int) (*SignatureResult, error) {
const bmpFileHeaderSize = 14
bmpHeaderFields := []StructField{
{"bfType", "u16"},
{"bfSize", "u32"},
{"bfReserved1", "u16"},
{"bfReserved2", "u16"},
{"bfOffBits", "u32"},
}
validDIBHeaderSizes := []int{12, 40, 108, 124}
bmpData := data[offset:]
if len(bmpData) < StructSize(bmpHeaderFields) {
return nil, fmt.Errorf("BMP: not enough data for header")
}
hdr, err := Parse(bmpData, bmpHeaderFields, "little")
if err != nil {
return nil, fmt.Errorf("BMP: parse error: %w", err)
}
bmpDataSize := len(bmpData)
// The BMP file size cannot be bigger than available data
if bmpDataSize < hdr["bfSize"] {
return nil, fmt.Errorf("BMP: reported size %d exceeds available data %d", hdr["bfSize"], bmpDataSize)
}
if hdr["bfSize"] == 0 {
return nil, fmt.Errorf("BMP: file size is 0")
}
if hdr["bfOffBits"] == 0 {
return nil, fmt.Errorf("BMP: bitmap bits offset is 0")
}
if hdr["bfOffBits"] > bmpDataSize {
return nil, fmt.Errorf("BMP: bitmap bits offset exceeds available data")
}
// Validate DIB header size
if offset+bmpFileHeaderSize+4 > len(data) {
return nil, fmt.Errorf("BMP: not enough data for DIB header size")
}
dibHeaderSize := int(LE32(data, offset+bmpFileHeaderSize))
validDIB := false
for _, vs := range validDIBHeaderSizes {
if dibHeaderSize == vs {
validDIB = true
break
}
}
if !validDIB {
return nil, fmt.Errorf("BMP: invalid DIB header size %d", dibHeaderSize)
}
// The offset that points to the image data cannot point into the second header
if hdr["bfOffBits"] < bmpFileHeaderSize+dibHeaderSize {
return nil, fmt.Errorf("BMP: bitmap bits offset points into header")
}
totalSize := hdr["bfSize"]
return &SignatureResult{
Offset: offset,
Name: "bmp",
Description: fmt.Sprintf("BMP image, total size: %d", totalSize),
Confidence: ConfidenceMedium,
Size: totalSize,
}, nil
}
// ---------------------------------------------------------------------------
// GIF
// ---------------------------------------------------------------------------
const GIFDescription = "GIF image"
func GIFMagic() [][]byte {
return [][]byte{
{0x47, 0x49, 0x46, 0x38, 0x37, 0x61}, // GIF87a
{0x47, 0x49, 0x46, 0x38, 0x39, 0x61}, // GIF89a
}
}
func GIFParser(data []byte, offset int) (*SignatureResult, error) {
gifHeaderFields := []StructField{
{"magic_p1", "u24"},
{"magic_p2", "u24"},
{"image_width", "u16"},
{"image_height", "u16"},
{"flags", "u8"},
{"bg_color_index", "u8"},
{"aspect_ratio", "u8"},
}
gifData := data[offset:]
if len(gifData) < StructSize(gifHeaderFields) {
return nil, fmt.Errorf("GIF: not enough data for header")
}
hdr, err := Parse(gifData, gifHeaderFields, "little")
if err != nil {
return nil, fmt.Errorf("GIF: parse error: %w", err)
}
// Calculate header size including optional global color table
headerSize := StructSize(gifHeaderFields) + gifColorTableSize(hdr["flags"])
if offset+headerSize > len(data) {
return nil, fmt.Errorf("GIF: not enough data for header + color table")
}
// Walk GIF blocks to find terminator
blockData := data[offset+headerSize:]
gifDataSize, err := getGIFDataSize(blockData)
if err != nil {
return nil, fmt.Errorf("GIF: %w", err)
}
totalSize := headerSize + gifDataSize
return &SignatureResult{
Offset: offset,
Description: fmt.Sprintf("%s, %dx%d pixels, total size: %d bytes", GIFDescription, hdr["image_width"], hdr["image_height"], totalSize),
Confidence: ConfidenceHigh,
Size: totalSize,
}, nil
}
func gifColorTableSize(flags int) int {
const hasColorTable = 0x80
const colorTableSizeMask = 0x07
if flags&hasColorTable == 0 {
return 0
}
encodedSize := (flags & colorTableSizeMask) + 1
size := 3
for i := 0; i < encodedSize; i++ {
size *= 2
}
return size
}
func getGIFDataSize(gifData []byte) (int, error) {
const extensionBlock byte = 0x21
const terminatorBlock byte = 0x3B
const imageDescriptorBlock byte = 0x2C
nextOffset := 0
var previousOffset *int
available := len(gifData)
for IsOffsetSafe(available, nextOffset, previousOffset) {
if nextOffset >= available {
break
}
blockType := gifData[nextOffset]
var blockSize int
var err error
switch blockType {
case imageDescriptorBlock:
blockSize, err = parseGIFImageDescriptor(gifData[nextOffset:])
case extensionBlock:
blockSize, err = parseGIFExtension(gifData[nextOffset:])
case terminatorBlock:
return nextOffset + 1, nil
default:
return 0, fmt.Errorf("unknown GIF block type 0x%02X at offset %d", blockType, nextOffset)
}
if err != nil {
return 0, err
}
prev := nextOffset
previousOffset = &prev
nextOffset += blockSize
}
return 0, fmt.Errorf("failed to find GIF terminator block")
}
func parseGIFImageDescriptor(gifData []byte) (int, error) {
const lzwCodeSize = 1
imgDescFields := []StructField{
{"magic", "u8"},
{"image_left", "u16"},
{"image_top", "u16"},
{"image_width", "u16"},
{"image_height", "u16"},
{"flags", "u8"},
}
if len(gifData) < StructSize(imgDescFields) {
return 0, fmt.Errorf("GIF image descriptor: not enough data")
}
hdr, err := Parse(gifData, imgDescFields, "little")
if err != nil {
return 0, err
}
totalSize := StructSize(imgDescFields) + gifColorTableSize(hdr["flags"]) + lzwCodeSize
if totalSize > len(gifData) {
return 0, fmt.Errorf("GIF image descriptor: not enough data for sub-blocks")
}
subBlocksSize, err := parseGIFSubBlocks(gifData[totalSize:])
if err != nil {
return 0, err
}
totalSize += subBlocksSize
return totalSize, nil
}
func parseGIFExtension(extensionData []byte) (int, error) {
const plainText = 1
const application = 0xFF
const headerSize = 2
extFields := []StructField{
{"magic", "u8"},
{"extension_type", "u8"},
{"sub_block_offset", "u8"},
}
if len(extensionData) < StructSize(extFields) {
return 0, fmt.Errorf("GIF extension: not enough data")
}
hdr, err := Parse(extensionData, extFields, "little")
if err != nil {
return 0, err
}
subBlocksOffset := headerSize
extType := hdr["extension_type"]
if extType == application || extType == plainText {
subBlocksOffset += hdr["sub_block_offset"] + 1
}
if subBlocksOffset > len(extensionData) {
return 0, fmt.Errorf("GIF extension: sub-blocks offset exceeds data")
}
subBlocksSize, err := parseGIFSubBlocks(extensionData[subBlocksOffset:])
if err != nil {
return 0, err
}
return subBlocksOffset + subBlocksSize, nil
}
func parseGIFSubBlocks(subBlockData []byte) (int, error) {
const subBlockTerminator byte = 0
available := len(subBlockData)
nextOffset := 0
var previousOffset *int
for IsOffsetSafe(available, nextOffset, previousOffset) {
if nextOffset >= available {
break
}
subBlockSize := subBlockData[nextOffset]
if subBlockSize == subBlockTerminator {
return nextOffset + 1, nil
}
prev := nextOffset
previousOffset = &prev
nextOffset += int(subBlockSize) + 1
}
return 0, fmt.Errorf("GIF sub-blocks: terminator not found")
}
// ---------------------------------------------------------------------------
// SVG
// ---------------------------------------------------------------------------
const SVGDescription = "SVG image"
const svgOpenTag = "<svg "
const svgCloseTag = "</svg>"
const svgHeadMagic = `xmlns="http://www.w3.org/2000/svg"`
func SVGMagic() [][]byte {
return [][]byte{
{0x3C, 0x73, 0x76, 0x67, 0x20}, // "<svg "
}
}
func SVGParser(data []byte, offset int) (*SignatureResult, error) {
// Operate directly on the byte slice. A previous implementation did
// string(data[offset:]) — copying the entire remaining file — on every
// call; since the parser runs once per "<svg " match, a buffer dense with
// "<svg " tokens made this O(n^2) in copies and could hang indefinitely.
svgData := data[offset:]
openTag := []byte(svgOpenTag)
closeTag := []byte(svgCloseTag)
headMagic := []byte(svgHeadMagic)
// The matched "<svg " must be a real SVG root: its opening tag has to
// terminate with '>' and declare the SVG xmlns (which the loop below
// requires anyway via headTagCount). Verifying this up front, with a cap
// on how far we look for the closing '>', keeps the parser O(1) per call
// regardless of file size — so a buffer densely packed with "<svg " tokens
// can't turn scanning into an O(n^2) hang.
const maxRootTagLen = 64 * 1024
rootLimit := len(svgData)
if rootLimit > maxRootTagLen {
rootLimit = maxRootTagLen
}
rootEnd := bytes.IndexByte(svgData[:rootLimit], '>')
if rootEnd < 0 {
return nil, fmt.Errorf("SVG: root tag not terminated")
}
if !bytes.Contains(svgData[:rootEnd+1], headMagic) {
return nil, fmt.Errorf("SVG: root tag missing SVG namespace")
}
headTagCount := 0
unclosedSVGTags := 0
// Search for all <svg and </svg> occurrences
searchIdx := 0
for searchIdx < len(svgData) {
// Find next occurrence of either tag
openIdx := bytes.Index(svgData[searchIdx:], openTag)
closeIdx := bytes.Index(svgData[searchIdx:], closeTag)
var tagStart int
if openIdx == -1 && closeIdx == -1 {
break
} else if openIdx == -1 {
tagStart = searchIdx + closeIdx
} else if closeIdx == -1 {
tagStart = searchIdx + openIdx
} else if openIdx <= closeIdx {
tagStart = searchIdx + openIdx
} else {
tagStart = searchIdx + closeIdx
}
tagBytes := svgData[tagStart:]
// Find the end '>' of this tag
endIdx := bytes.IndexByte(tagBytes, '>')
if endIdx == -1 {
break
}
fullTag := tagBytes[:endIdx+1]
isOpen := bytes.HasPrefix(fullTag, openTag)
isClose := bytes.HasPrefix(fullTag, closeTag)
isHead := bytes.Contains(fullTag, headMagic)
if isHead {
headTagCount++
}
if isOpen {
unclosedSVGTags++
}
if isClose {
unclosedSVGTags--
}
// More than one head tag is invalid
if headTagCount > 1 {
break
}
// If one head tag and all tags closed, that's EOF
if headTagCount == 1 && unclosedSVGTags == 0 {
totalSize := tagStart + len(svgCloseTag)
return &SignatureResult{
Offset: offset,
Description: fmt.Sprintf("%s, total size: %d bytes", SVGDescription, totalSize),
Confidence: ConfidenceMedium,
Size: totalSize,
}, nil
}
searchIdx = tagStart + 1
}
return nil, fmt.Errorf("SVG: failed to find matching closing tag")
}
// ---------------------------------------------------------------------------
// RIFF
// ---------------------------------------------------------------------------
const RIFFDescription = "RIFF image"
func RIFFMagic() [][]byte {
return [][]byte{
{0x52, 0x49, 0x46, 0x46}, // "RIFF"
}
}
func RIFFParser(data []byte, offset int) (*SignatureResult, error) {
const riffMagicValue = 0x46464952 // "RIFF" in LE
const chunkTypeStart = 8
const chunkTypeEnd = 12
const fileSizeOffset = 8
riffFields := []StructField{
{"magic", "u32"},
{"file_size", "u32"},
{"chunk_type", "u32"},
}
riffData := data[offset:]
if len(riffData) < StructSize(riffFields) {
return nil, fmt.Errorf("RIFF: not enough data for header")
}
hdr, err := Parse(riffData, riffFields, "little")
if err != nil {
return nil, fmt.Errorf("RIFF: parse error: %w", err)
}
if hdr["magic"] != riffMagicValue {
return nil, fmt.Errorf("RIFF: invalid magic")
}
if chunkTypeEnd > len(riffData) {
return nil, fmt.Errorf("RIFF: not enough data for chunk type string")
}
// Get the RIFF type string (e.g., "WAVE", "AVI ")
typeBytes := riffData[chunkTypeStart:chunkTypeEnd]
typeString := strings.TrimSpace(string(typeBytes))
// Validate that the type string is valid UTF-8 / printable
for _, b := range typeBytes {
if b < 0x20 || b > 0x7E {
return nil, fmt.Errorf("RIFF: non-printable chunk type")
}
}
totalSize := hdr["file_size"] + fileSizeOffset
return &SignatureResult{
Offset: offset,
Description: fmt.Sprintf("%s, encoding type: %s, total size: %d bytes", RIFFDescription, typeString, totalSize),
Confidence: ConfidenceMedium,
Size: totalSize,
}, nil
}
// ---------------------------------------------------------------------------
// PDF
// ---------------------------------------------------------------------------
const PDFDescription = "PDF document"
func PDFMagic() [][]byte {
return [][]byte{
{0x25, 0x50, 0x44, 0x46, 0x2D, 0x31, 0x2E}, // "%PDF-1."
}
}
func PDFParser(data []byte, offset int) (*SignatureResult, error) {
const minPDFSize = 16
const newlineOffset = 8
const minorNumberOffset = 7
const asciiZero byte = 0x30
const asciiNine byte = 0x39
const asciiNewline byte = 0x0A
const asciiPercent byte = 0x25
const asciiCarriageReturn byte = 0x0D
if offset+minPDFSize > len(data) {
return nil, fmt.Errorf("PDF: not enough data for header")
}
pdfHeader := data[offset : offset+minPDFSize]
// Get the minor version number
versionMinor := pdfHeader[minorNumberOffset]
if versionMinor < asciiZero || versionMinor > asciiNine {
return nil, fmt.Errorf("PDF: invalid minor version number")
}
// After the version line, expect newline(s) followed by '%'
for _, b := range pdfHeader[newlineOffset:] {
if b == asciiNewline || b == asciiCarriageReturn {
continue
} else if b == asciiPercent {
return &SignatureResult{
Offset: offset,
Description: fmt.Sprintf("%s, version 1.%d", PDFDescription, versionMinor-asciiZero),
Confidence: ConfidenceLow,
Size: 0,
}, nil
} else {
break
}
}
return nil, fmt.Errorf("PDF: invalid header format")
}
// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
func byteInSlice(b byte, s []byte) bool {
return bytes.ContainsRune(s, rune(b))
}
File diff suppressed because it is too large Load Diff
+117
View File
@@ -0,0 +1,117 @@
package binsnoop
import (
"crypto/rand"
"fmt"
"sort"
)
// Confidence levels for SignatureResult
const (
ConfidenceLow uint8 = 0
ConfidenceMedium uint8 = 128
ConfidenceHigh uint8 = 250
)
// SignatureResult describes a validated file signature found during scanning.
type SignatureResult struct {
Offset int `json:"offset"`
ID string `json:"id"`
Size int `json:"size"`
Name string `json:"name"`
Confidence uint8 `json:"confidence"`
Description string `json:"description"`
AlwaysDisplay bool `json:"always_display"`
ExtractionDeclined bool `json:"extraction_declined"`
PreferredExtractor *Extractor `json:"-"`
}
// DeepScanResult is a validated signature retained by a non-suppressing scan.
// ParentOffset is -1 for a top-level hit.
type DeepScanResult struct {
SignatureResult
End int `json:"end"`
Depth int `json:"depth"`
ParentOffset int `json:"parent_offset"`
Overlap bool `json:"overlap"`
}
// DeepScan is a bounded nested-signature analysis result.
type DeepScan struct {
Results []DeepScanResult `json:"results"`
Truncated bool `json:"truncated"`
}
// SignatureParser validates a magic byte match and returns a SignatureResult or error.
type SignatureParser func(data []byte, offset int) (*SignatureResult, error)
// Signature defines a file signature to search for.
type Signature struct {
Name string
Short bool
Magic [][]byte
MagicOffset int
Description string
AlwaysDisplay bool
Parser SignatureParser
Extractor *Extractor
}
// ExtractorType defines the type of extractor.
type ExtractorType int
const (
ExtractorTypeNone ExtractorType = iota
ExtractorTypeInternal
ExtractorTypeExternal
)
// SourceFilePlaceholder is replaced at runtime with the actual file path.
const SourceFilePlaceholder = "%e"
// InternalExtractorFunc is the function signature for built-in extractors.
type InternalExtractorFunc func(data []byte, offset int, outputDir string) *ExtractionResult
// Extractor describes how to extract a file type.
type Extractor struct {
Type ExtractorType
Command string
Function InternalExtractorFunc
Extension string
Arguments []string
ExitCodes []int
DoNotRecurse bool
}
// ExtractionResult stores information about a completed extraction.
type ExtractionResult struct {
Size *int `json:"size,omitempty"`
Success bool `json:"success"`
ExtractorName string `json:"extractor"`
DoNotRecurse bool `json:"do_not_recurse"`
OutputDirectory string `json:"output_directory"`
}
// AnalysisResults contains all results from analysing a file or buffer.
type AnalysisResults struct {
FilePath string `json:"file_path"`
FileMap []SignatureResult `json:"file_map"`
Extractions map[string]*ExtractionResult `json:"extractions"`
}
// sortSignatureResults sorts by offset for proper file mapping.
func sortSignatureResults(results []SignatureResult) {
sort.Slice(results, func(i, j int) bool {
return results[i].Offset < results[j].Offset
})
}
// newUUID generates a v4 UUID string with no external dependencies.
func newUUID() string {
b := make([]byte, 16)
_, _ = rand.Read(b)
b[6] = (b[6] & 0x0f) | 0x40
b[8] = (b[8] & 0x3f) | 0x80
return fmt.Sprintf("%08x-%04x-%04x-%04x-%012x",
b[0:4], b[4:6], b[6:8], b[8:10], b[10:16])
}