278 lines
9.7 KiB
Markdown
278 lines
9.7 KiB
Markdown
# binsnoop
|
|
|
|
A pure-Go binary analysis library for detecting and extracting embedded files, firmware images, compressed data, filesystems, and more from binary blobs. Ported from the [binwalk](https://github.com/ReFirmLabs/binwalk) Rust codebase with like-for-like scanning logic.
|
|
|
|
**Zero external dependencies** — everything is self-contained, including the Aho-Corasick multi-pattern matcher, CRC32 validation, and binary structure parsing.
|
|
|
|
## Features
|
|
|
|
- **111 built-in signatures** covering archives, compression, filesystems, images, executables, firmware headers, cryptographic constructs, and more
|
|
- **Aho-Corasick** multi-pattern scanning with overlapping match support
|
|
- **Signature validation** — every magic byte match is verified by a dedicated parser that checks headers, checksums, and structural integrity
|
|
- **Confidence scoring** — results are ranked Low / Medium / High so callers can filter noise
|
|
- **Extraction support** — carve identified objects out of a binary, or invoke external tools (7z, unsquashfs, etc.) to fully extract them
|
|
- **Filtering** — include/exclude signatures by name, restrict short signatures to offset 0, or add custom signatures
|
|
- **JSON-friendly** — all result types carry `json` struct tags for easy serialisation
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
go get binsnoop
|
|
```
|
|
|
|
Or add to your `go.mod`:
|
|
|
|
```
|
|
require binsnoop v0.0.0
|
|
```
|
|
|
|
## Quick Start
|
|
|
|
```go
|
|
package main
|
|
|
|
import (
|
|
"fmt"
|
|
"binsnoop"
|
|
)
|
|
|
|
func main() {
|
|
scanner := binsnoop.New()
|
|
|
|
// Scan a file
|
|
results, err := scanner.ScanFile("firmware.bin")
|
|
if err != nil {
|
|
panic(err)
|
|
}
|
|
|
|
for _, r := range results {
|
|
fmt.Printf("[0x%08X] %s (confidence: %d, size: %d)\n",
|
|
r.Offset, r.Description, r.Confidence, r.Size)
|
|
}
|
|
}
|
|
```
|
|
|
|
## API Reference
|
|
|
|
### Creating a Scanner
|
|
|
|
```go
|
|
// Default scanner with all 111 signatures
|
|
scanner := binsnoop.New()
|
|
|
|
// Scanner with options
|
|
scanner, err := binsnoop.NewWithOptions(binsnoop.Options{
|
|
Include: []string{"gzip", "squashfs", "uimage"}, // only these signatures
|
|
Exclude: []string{"copyright"}, // remove these signatures
|
|
FullSearch: true, // match short sigs at any offset
|
|
AdditionalSignatures: []binsnoop.Signature{...}, // add custom signatures
|
|
})
|
|
```
|
|
|
|
### Scanning
|
|
|
|
```go
|
|
// Scan raw bytes
|
|
results := scanner.ScanBytes(data)
|
|
|
|
// Scan a file on disk
|
|
results, err := scanner.ScanFile("/path/to/firmware.bin")
|
|
```
|
|
|
|
Both return `[]SignatureResult` sorted by offset.
|
|
|
|
### Analysis (Scan + Extraction)
|
|
|
|
```go
|
|
// Analyse raw bytes (with optional extraction)
|
|
analysis := scanner.AnalyzeBytes(data, "firmware.bin", true)
|
|
|
|
// Analyse a file on disk
|
|
analysis, err := scanner.AnalyzeFile("/path/to/firmware.bin", true)
|
|
```
|
|
|
|
Returns `*AnalysisResults` containing the file map and extraction outcomes.
|
|
|
|
### Carving
|
|
|
|
Extract a single identified object from raw data:
|
|
|
|
```go
|
|
err := binsnoop.CarveResult(data, results[0], "/tmp/output")
|
|
```
|
|
|
|
This writes the bytes at the result's offset/size to a file in the output directory.
|
|
|
|
### Listing Signatures
|
|
|
|
```go
|
|
names := binsnoop.ListSignatures() // []string of all registered signature names
|
|
```
|
|
|
|
### Scanner Info
|
|
|
|
```go
|
|
scanner.SignatureCount() // number of registered signatures (111)
|
|
scanner.PatternCount() // total magic byte patterns across all signatures
|
|
```
|
|
|
|
## Types
|
|
|
|
### `SignatureResult`
|
|
|
|
```go
|
|
type SignatureResult struct {
|
|
Offset int `json:"offset"` // byte offset in the input data
|
|
ID string `json:"id"` // unique identifier for this match
|
|
Size int `json:"size"` // size in bytes (0 = unknown)
|
|
Name string `json:"name"` // signature name (e.g. "gzip")
|
|
Confidence uint8 `json:"confidence"` // 0=Low, 128=Medium, 250=High
|
|
Description string `json:"description"` // human-readable description
|
|
AlwaysDisplay bool `json:"always_display"` // show even if low confidence
|
|
ExtractionDeclined bool `json:"extraction_declined"` // parser advises against extraction
|
|
PreferredExtractor *Extractor `json:"-"` // extractor to use for this result
|
|
}
|
|
```
|
|
|
|
### `Options`
|
|
|
|
```go
|
|
type Options struct {
|
|
Include []string // restrict to these signature names
|
|
Exclude []string // remove these signature names
|
|
FullSearch bool // short sigs match at any offset
|
|
AdditionalSignatures []Signature // user-defined signatures
|
|
}
|
|
```
|
|
|
|
### `AnalysisResults`
|
|
|
|
```go
|
|
type AnalysisResults struct {
|
|
FilePath string `json:"file_path"`
|
|
FileMap []SignatureResult `json:"file_map"`
|
|
Extractions map[string]*ExtractionResult `json:"extractions"`
|
|
}
|
|
```
|
|
|
|
### Confidence Levels
|
|
|
|
| Constant | Value | Meaning |
|
|
|----------|-------|---------|
|
|
| `ConfidenceLow` | 0 | Magic matched but weak validation |
|
|
| `ConfidenceMedium` | 128 | Header parsed and mostly valid |
|
|
| `ConfidenceHigh` | 250 | Full structural validation passed |
|
|
|
|
## Custom Signatures
|
|
|
|
You can add your own signatures at scanner creation:
|
|
|
|
```go
|
|
mySignature := binsnoop.Signature{
|
|
Name: "my_format",
|
|
Magic: [][]byte{[]byte("MYMAGIC")},
|
|
Parser: func(data []byte, offset int) (*binsnoop.SignatureResult, error) {
|
|
// Validate the match and return a result
|
|
if offset+16 > len(data) {
|
|
return nil, fmt.Errorf("insufficient data")
|
|
}
|
|
return &binsnoop.SignatureResult{
|
|
Offset: offset,
|
|
Size: 1024,
|
|
Description: "My custom format, size: 1024 bytes",
|
|
Confidence: binsnoop.ConfidenceHigh,
|
|
}, nil
|
|
},
|
|
Description: "My custom file format",
|
|
}
|
|
|
|
scanner, _ := binsnoop.NewWithOptions(binsnoop.Options{
|
|
AdditionalSignatures: []binsnoop.Signature{mySignature},
|
|
})
|
|
```
|
|
|
|
## Supported Signatures
|
|
|
|
### Archives (12)
|
|
gzip, zip, tarball, 7zip, rar, cab, deb, cpio, arj, iso9660, dahua\_zip
|
|
|
|
### Compression (10)
|
|
lzma, bzip2, xz, zstd, lz4, lzop, lzfse, zlib, compress'd
|
|
|
|
### Filesystems (13)
|
|
squashfs, cramfs, jffs2, yaffs, romfs, ext, fat, ntfs, apfs, btrfs, logfs, ubi, ubifs
|
|
|
|
### Images & Media (7)
|
|
png, jpeg, bmp, gif, svg, riff, pdf
|
|
|
|
### Executables (2)
|
|
elf, pe
|
|
|
|
### Firmware (32)
|
|
uimage, uboot, trx, tplink, tplink\_rtos, dlob, arcadyan, chk, seama, cfe, jboot\_arm, jboot\_stag, jboot\_sch2, packimg, binhdr, autel, android\_sparse, android\_bootimg, dtb, dlink\_tlv, dlke, shrs, dkbs, dms, encfw, encrpted\_img, wince, mh01, csman, dxbc, matter\_ota, rtk
|
|
|
|
### Cryptographic (17)
|
|
aes\_sbox, aes\_forward\_table, aes\_reverse\_table, aes\_rcon, aes\_acceleration\_table, rsa, openssl, gpg\_signed, pem\_certificate, pem\_public\_key, pem\_private\_key, pkcs\_der\_hash, luks, crc32, sha256, md5
|
|
|
|
### Miscellaneous (18)
|
|
copyright, wind\_kernel, vxworks\_symtab, ecos, linux\_kernel, linux\_boot\_image, linux\_arm\_zimage, linux\_arm64\_boot\_image, efigpt, mbr, uefi\_pi\_volume, uefi\_capsule, pchrom, pcapng, pjl, srecord, dmg, qnx\_ifs, dpapi, qcow
|
|
|
|
## Extraction
|
|
|
|
Many signatures have associated extractors that invoke external command-line tools. When you call `AnalyzeFile` or `AnalyzeBytes` with `extract: true`, binsnoop will:
|
|
|
|
1. Carve the identified data to a temporary file
|
|
2. Run the configured external tool (e.g. `7z x`, `unsquashfs`, `jefferson`)
|
|
3. Return the extraction results including output directory and success/failure status
|
|
|
|
**Common external tools used:**
|
|
|
|
| Tool | Signatures |
|
|
|------|-----------|
|
|
| `7z` | zip, cpio, arj, iso9660, xz, lzma, squashfs, dmg, efigpt |
|
|
| `gzip` | gzip |
|
|
| `bzip2` | bzip2 |
|
|
| `zstd` | zstd |
|
|
| `lz4` | lz4 |
|
|
| `unsquashfs` | squashfs |
|
|
| `jefferson` | jffs2 |
|
|
| `unyaffs` | yaffs |
|
|
| `tsk_recover` | ext, fat, ntfs |
|
|
| `uefi-firmware-parser` | uefi\_pi\_volume, uefi\_capsule, pchrom |
|
|
| `simg2img` | android\_sparse |
|
|
| `dtc` | dtb |
|
|
|
|
## How It Works
|
|
|
|
### Scanning Algorithm
|
|
|
|
1. **Short-signature pass** — signatures marked `Short` (e.g. zlib, MBR) are checked only at offset 0 (unless `FullSearch` is enabled)
|
|
2. **Aho-Corasick scan** — all magic byte patterns are compiled into an Aho-Corasick automaton for efficient multi-pattern matching. When a high-confidence sized match is found, scanning restarts from the end of that match
|
|
3. **Validation** — every magic match is passed to its signature parser which validates headers, checksums, and structure
|
|
4. **Post-processing** — results are sorted by offset, overlapping/duplicate matches are resolved (higher confidence wins), and unknown sizes are estimated based on the next significant match or EOF
|
|
|
|
### Architecture
|
|
|
|
```
|
|
binsnoop.go Public API (New, ScanBytes, ScanFile, AnalyzeFile, CarveResult, etc.)
|
|
scanner.go Core scan engine (Aho-Corasick integration, post-processing)
|
|
match.go Self-contained Aho-Corasick trie with overlapping match support
|
|
parse.go Generic binary structure parser (u8/u16/u24/u32/u64, big/little endian)
|
|
types.go Core types (SignatureResult, Signature, Extractor, Options, etc.)
|
|
common.go Utility functions (CRC32, endian helpers, string helpers)
|
|
extract.go Carving and external tool execution
|
|
magic.go Signature registry — wires all 111 signatures together
|
|
sig_archives.go Archive signature parsers
|
|
sig_compression.go Compression signature parsers
|
|
sig_filesystems.go Filesystem signature parsers
|
|
sig_images.go Image/media signature parsers
|
|
sig_executables.go Executable signature parsers
|
|
sig_firmware.go Firmware header signature parsers
|
|
sig_crypto.go Cryptographic signature parsers
|
|
sig_misc.go Miscellaneous signature parsers
|
|
```
|
|
|
|
## License
|
|
|
|
This project is a Go port of [binwalk](https://github.com/ReFirmLabs/binwalk) by ReFirmLabs. See the original project for licensing details.
|