Skip to content

Stream acquisitions to zip archives #110

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
besendorf wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Stream acquisitions to zip archives #110

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6831ac4
Stream acquisitions to zip archives
besendorf Jun 23, 2026
0554271
Fix APK archive collection regressions
besendorf Jun 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ Before launching androidqf you need to have the target Android device connected

Once USB debugging is enabled, you can proceed launching androidqf. It will first attempt to connect to the device over the USB bridge, which should result in the Android phone to prompt you to manually authorize the host keys. Make sure to authorize them, ideally permanently so that the prompt wouldn't appear again.

Now androidqf should be executing and creating an acquisition folder in your current working directory. At some point in the execution, androidqf will prompt you some choices: these prompts will pause the acquisition until you provide a selection, so pay attention.
Now androidqf should be executing and creating an acquisition zip archive in your current working directory, or in the directory provided with `-output`. At some point in the execution, androidqf will prompt you some choices: these prompts will pause the acquisition until you provide a selection, so pay attention.

The following data can be extracted:

Expand Down Expand Up @@ -186,7 +186,7 @@ Ideally you should have the drive fully encrypted, but that might not always be

Alternatively, androidqf allows to encrypt each acquisition with a provided [age](https://age-encryption.org) public key. Preferably, this public key belongs to a keypair for which the end-user does not possess, or at least carry, the private key. In this way, the end-user would not be able to decrypt the acquired data even under duress.

If you place a file called `key.txt` in the current working directory, androidqf will automatically attempt to compress and encrypt each acquisition and delete the original unencrypted copies. androidqf also checks for `key.txt` in the same folder as the executable; if both files exist, the current working directory takes precedence.
androidqf streams each acquisition into a zip archive. If you place a file called `key.txt` in the current working directory, androidqf will encrypt the zip stream with age and write `<UUID>.zip.age`; otherwise, it writes an unencrypted `<UUID>.zip`. androidqf also checks for `key.txt` in the same folder as the executable; if both files exist, the current working directory takes precedence.

Once you have retrieved an encrypted acquisition file, you can decrypt it with age like so:

Expand Down
210 changes: 51 additions & 159 deletions acquisition/acquisition.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,23 +7,22 @@ package acquisition

import (
"bytes"
"encoding/csv"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"time"

"github.com/botherder/go-savetime/hashes"
"github.com/google/uuid"
"github.com/mvt-project/androidqf/adb"
"github.com/mvt-project/androidqf/assets"
"github.com/mvt-project/androidqf/log"
"github.com/mvt-project/androidqf/utils"
)

const streamingPullerMemoryLimitMB = 500

// Acquisition is the main object containing all phone information
type Acquisition struct {
UUID string `json:"uuid"`
Expand All @@ -36,7 +35,7 @@ type Acquisition struct {
SdCard string `json:"sdcard"`
Cpu string `json:"cpu"`
closeLog func() `json:"-"`
EncryptedWriter *EncryptedZipWriter `json:"-"`
ZipWriter *StreamingZipWriter `json:"-"`
StreamingMode bool `json:"streaming_mode"`
StreamingPuller *StreamingPuller `json:"-"`
logBuffer *bytes.Buffer `json:"-"`
Expand All @@ -48,28 +47,11 @@ func New(path string) (*Acquisition, error) {
UUID: uuid.New().String(),
Started: time.Now().UTC(),
AndroidQFVersion: utils.Version,
}

if path == "" {
acq.StoragePath = acq.UUID
} else {
acq.StoragePath = path
}
// Check if the path exist
stat, err := os.Stat(acq.StoragePath)
if os.IsNotExist(err) {
err := os.Mkdir(acq.StoragePath, 0o755)
if err != nil {
return nil, fmt.Errorf("failed to create acquisition folder: %v", err)
}
} else {
if !stat.IsDir() {
return nil, fmt.Errorf("path exist and is not a folder")
}
StreamingMode: true,
}

// Get system information first to get tmp folder
err = acq.GetSystemInformation()
err := acq.GetSystemInformation()
if err != nil {
return nil, err
}
Expand All @@ -81,39 +63,24 @@ func New(path string) (*Acquisition, error) {
}
acq.Collector = coll

// Try to initialize encrypted streaming mode
encWriter, err := NewEncryptedZipWriter(acq.UUID)
zipWriter, err := NewStreamingZipWriter(acq.UUID, path)
if err != nil {
// No key file or encryption setup failed, use normal mode
log.Debug("Encrypted streaming not available, using normal mode")
acq.StreamingMode = false

// Init logging file for normal mode
logPath := filepath.Join(acq.StoragePath, "command.log")
closeLog, err := log.EnableFileLog(log.DEBUG, logPath)
if err != nil {
return nil, fmt.Errorf("failed to enable file logging: %v", err)
}
acq.closeLog = closeLog
} else {
// Encrypted streaming mode enabled
log.Info("Using encrypted streaming mode - data will be written directly to encrypted archive")
acq.StreamingMode = true
acq.EncryptedWriter = encWriter
return nil, err
}
acq.ZipWriter = zipWriter
acq.StoragePath = zipWriter.GetOutputPath()

// Initialize streaming puller for direct operations
acq.StreamingPuller = NewStreamingPuller(adb.Client.ExePath, adb.Client.Serial, 100) // 100MB max memory
// Initialize streaming puller for direct operations.
acq.StreamingPuller = NewStreamingPuller(adb.Client.ExePath, adb.Client.Serial, streamingPullerMemoryLimitMB)

// Create buffer for command.log (will be written to archive at completion)
acq.logBuffer = new(bytes.Buffer)
// Create buffer for command.log (will be written to archive at completion).
acq.logBuffer = new(bytes.Buffer)

// Init logging to write to buffer
closeLog, err := log.EnableWriterLog(log.DEBUG, acq.logBuffer)
if err != nil {
return nil, fmt.Errorf("failed to enable writer logging: %v", err)
}
acq.closeLog = closeLog
closeLog, err := log.EnableWriterLog(log.DEBUG, acq.logBuffer)
if err != nil {
return nil, fmt.Errorf("failed to enable writer logging: %v", err)
}
acq.closeLog = closeLog

return &acq, nil
}
Expand All @@ -123,16 +90,15 @@ func (a *Acquisition) Complete() {
a.Completed = time.Now().UTC()
}

// Handle streaming mode completion
if a.StreamingMode && a.EncryptedWriter != nil {
// Store acquisition info in the encrypted zip
if a.ZipWriter != nil {
// Store acquisition info in the zip
info, err := json.MarshalIndent(a, "", " ")
if err != nil {
log.Error("Failed to marshal acquisition info for encrypted archive")
log.Error("Failed to marshal acquisition info for archive")
} else {
err = a.EncryptedWriter.CreateFileFromBytes("acquisition.json", info)
err = a.ZipWriter.CreateFileFromBytes("acquisition.json", info)
if err != nil {
log.ErrorExc("Failed to store acquisition info in encrypted archive", err)
log.ErrorExc("Failed to store acquisition info in archive", err)
}
}

Expand All @@ -142,33 +108,22 @@ func (a *Acquisition) Complete() {
a.closeLog()
}

// Write buffered command.log to encrypted archive
// Write buffered command.log to archive
if a.logBuffer != nil && a.logBuffer.Len() > 0 {
err = a.EncryptedWriter.CreateFileFromBytes("command.log", a.logBuffer.Bytes())
err = a.ZipWriter.CreateFileFromBytes("command.log", a.logBuffer.Bytes())
if err != nil {
log.ErrorExc("Failed to add command.log to encrypted archive", err)
log.ErrorExc("Failed to add command.log to archive", err)
}
}

err = a.EncryptedWriter.CreateHashList()
err = a.ZipWriter.CreateHashList()
if err != nil {
log.ErrorExc("Failed to add hashes.csv to encrypted archive", err)
log.ErrorExc("Failed to add hashes.csv to archive", err)
}

// Close the encrypted writer
err = a.EncryptedWriter.Close()
err = a.ZipWriter.Close()
if err != nil {
log.ErrorExc("Failed to close encrypted archive", err)
}

// Remove the temporary storage directory if it was created and used
if a.StoragePath != "" {
if _, err := os.Stat(a.StoragePath); err == nil {
err = os.RemoveAll(a.StoragePath)
if err != nil {
log.ErrorExc("Failed to clean up temporary storage directory", err)
}
}
log.ErrorExc("Failed to close archive", err)
}
} else {
// Ensure log file is closed before cleanup operations
Expand Down Expand Up @@ -225,80 +180,7 @@ func (a *Acquisition) GetSystemInformation() error {
return nil
}

func (a *Acquisition) HashFiles() error {
// In streaming mode, files are directly encrypted and no local files exist to hash
if a.StreamingMode {
log.Debug("Skipping hash generation in streaming mode (data is encrypted)")
return nil
}

log.Info("Generating list of files hashes...")

csvFile, err := os.Create(filepath.Join(a.StoragePath, "hashes.csv"))
if err != nil {
return err
}
defer csvFile.Close()

csvWriter := csv.NewWriter(csvFile)

walkErr := filepath.Walk(a.StoragePath, func(filePath string, fileInfo os.FileInfo, err error) error {
if err != nil {
return err
}

if fileInfo.IsDir() {
return nil
}
// Makes files read only
os.Chmod(filePath, 0o400)

sha256, err := hashes.FileSHA256(filePath)
if err != nil {
return err
}

return csvWriter.Write([]string{filePath, sha256})
})

csvWriter.Flush()
if err := csvWriter.Error(); err != nil {
return err
}

return walkErr
}

func (a *Acquisition) StoreInfo() error {
// In streaming mode, info is stored during Complete()
if a.StreamingMode {
return nil
}

if a.Completed.IsZero() {
a.Completed = time.Now().UTC()
}

log.Info("Saving details about acquisition and device...")

info, err := json.MarshalIndent(a, "", " ")
if err != nil {
return fmt.Errorf("failed to json marshal the acquisition details: %v",
err)
}

infoPath := filepath.Join(a.StoragePath, "acquisition.json")

err = os.WriteFile(infoPath, info, 0o644)
if err != nil {
return fmt.Errorf("failed to write acquisition details to file: %v",
err)
}

return nil
}

// StreamAPKToZip streams an APK file directly to encrypted zip with certificate processing
// StreamAPKToZip streams an APK file directly to the zip with certificate processing
func (a *Acquisition) StreamAPKToZip(remotePath, zipPath string, processFunc func(io.Reader) error) error {
if err := a.validateStreamingMode(); err != nil {
return err
Expand All @@ -314,6 +196,17 @@ func (a *Acquisition) StreamAPKToZip(remotePath, zipPath string, processFunc fun
// Pull APK data to memory buffer
buffer, err := a.StreamingPuller.PullToBuffer(remotePath)
if err != nil {
if errors.Is(err, ErrStreamingBufferMemoryLimit) && processFunc == nil {
log.Debugf("APK %s exceeded streaming buffer limit; streaming directly to archive", remotePath)
writer, err := a.ZipWriter.CreateFile(zipPath)
if err != nil {
return fmt.Errorf("failed to create zip entry for APK %q: %v", remotePath, err)
}
if err := a.StreamingPuller.PullToWriter(remotePath, writer); err != nil {
return fmt.Errorf("failed to stream APK %q to zip: %v", remotePath, err)
}
return nil
}
return fmt.Errorf("failed to pull APK %q: %v", remotePath, err)
}

Expand All @@ -325,16 +218,15 @@ func (a *Acquisition) StreamAPKToZip(remotePath, zipPath string, processFunc fun
}
}

// Stream to encrypted zip
err = a.EncryptedWriter.CreateFileFromReader(zipPath, buffer.Reader())
err = a.ZipWriter.CreateFileFromReader(zipPath, buffer.Reader())
if err != nil {
return fmt.Errorf("failed to add APK %q to encrypted zip: %v", remotePath, err)
return fmt.Errorf("failed to add APK %q to zip: %v", remotePath, err)
}

return nil
}

// StreamBackupToZip streams a backup directly to encrypted zip
// StreamBackupToZip streams a backup directly to the zip
func (a *Acquisition) StreamBackupToZip(arg, zipPath string) error {
if err := a.validateStreamingMode(); err != nil {
return err
Expand All @@ -348,7 +240,7 @@ func (a *Acquisition) StreamBackupToZip(arg, zipPath string) error {
}

// Create zip entry writer
writer, err := a.EncryptedWriter.CreateFile(zipPath)
writer, err := a.ZipWriter.CreateFile(zipPath)
if err != nil {
return fmt.Errorf("failed to create zip entry for backup: %v", err)
}
Expand All @@ -362,7 +254,7 @@ func (a *Acquisition) StreamBackupToZip(arg, zipPath string) error {
return nil
}

// StreamBugreportToZip streams a bugreport directly to encrypted zip
// StreamBugreportToZip streams a bugreport directly to the zip
func (a *Acquisition) StreamBugreportToZip(zipPath string) error {
if err := a.validateStreamingMode(); err != nil {
return err
Expand All @@ -373,7 +265,7 @@ func (a *Acquisition) StreamBugreportToZip(zipPath string) error {
}

// Create zip entry writer
writer, err := a.EncryptedWriter.CreateFile(zipPath)
writer, err := a.ZipWriter.CreateFile(zipPath)
if err != nil {
return fmt.Errorf("failed to create zip entry for bugreport: %v", err)
}
Expand All @@ -392,8 +284,8 @@ func (a *Acquisition) validateStreamingMode() error {
if !a.StreamingMode {
return fmt.Errorf("streaming mode not enabled")
}
if a.EncryptedWriter == nil {
return fmt.Errorf("encrypted writer not initialized")
if a.ZipWriter == nil {
return fmt.Errorf("zip writer not initialized")
}
if a.StreamingPuller == nil {
return fmt.Errorf("streaming puller not initialized")
Expand Down
Loading
Loading