# RFC-040 Implementation: Batch Client

This document describes the implementation of RFC-040: fishinfo-python-sdk Batch Client.

## Overview

RFC-040 adds auto-batching capabilities to the fishinfo Python SDK, extending RFC-039 with:
- Automatic batching based on size and time thresholds
- Multiple buffer implementations (memory and disk)
- Compression support for large payloads
- Retry logic with exponential backoff
- Dead letter queue for permanently failed points
- Context manager support

## Architecture

### Directory Structure

```
sdk/python/fishinfo/src/fishinfo/
├── ingestion/
│   ├── __init__.py
│   └── batch.py              # BatchIngestionClient
├── buffer/
│   ├── __init__.py           # Buffer protocol
│   ├── memory.py             # In-memory buffer
│   └── disk.py               # Disk-based buffer (SQLite)
├── compression.py            # Compression utilities
├── retry.py                  # Retry logic and DLQ
└── exceptions.py             # Exception types
```

## Components

### 1. BatchIngestionClient (`ingestion/batch.py`)

The main auto-batching client for telemetry ingestion.

**Features:**
- Auto-flush on batch size threshold (default: 100 points)
- Auto-flush on batch age threshold (default: 5.0 seconds)
- Manual flush support
- Retry with exponential backoff
- Optional offline buffer
- Optional compression
- Statistics tracking
- Callback support
- Context manager support

**Usage:**

```python
from fishinfo import FishinfoClient
from fishinfo.ingestion import BatchIngestionClient
from fishinfo.buffer.memory import MemoryBuffer

# Create main client
client = FishinfoClient(config)
await client.connect()

# Create batch client with memory buffer
buffer = MemoryBuffer(max_size=5000)

async with BatchIngestionClient(
    client=client,
    max_batch_size=100,
    max_batch_age=5.0,
    buffer=buffer,
    compression=True,
) as batch_client:
    # Add points (non-blocking)
    await batch_client.add(point1)
    await batch_client.add(point2)

    # Points are automatically flushed based on thresholds
    # Manual flush if needed
    response = await batch_client.flush()

    # Get statistics
    stats = batch_client.get_stats()
    print(f"Sent: {stats.points_sent}, Failed: {stats.points_failed}")

# Automatically flushes and stops on context exit
```

### 2. Buffer Protocol (`buffer/__init__.py`)

Defines the interface for buffer implementations:

```python
class Buffer(Protocol):
    async def push(self, points: list[TelemetryPoint]) -> None: ...
    async def pop(self, max_count: int) -> list[TelemetryPoint]: ...
    async def peek(self, max_count: int) -> list[TelemetryPoint]: ...
    async def count(self) -> int: ...
    async def clear(self) -> None: ...
```

### 3. MemoryBuffer (`buffer/memory.py`)

In-memory buffer using `asyncio.Queue`:
- Fast, thread-safe
- Data lost on process restart
- Configurable max size
- FIFO ordering

**Usage:**

```python
from fishinfo.buffer.memory import MemoryBuffer

buffer = MemoryBuffer(max_size=10000)

# Push points
await buffer.push(points)

# Pop points (removes them)
popped = await buffer.pop(100)

# Peek without removing
peeked = await buffer.peek(100)

# Check count
count = await buffer.count()

# Clear all
await buffer.clear()
```

### 4. DiskBuffer (`buffer/disk.py`)

SQLite-based persistent buffer:
- Survives process restarts
- Automatic cleanup with VACUUM
- Configurable max size
- FIFO ordering

**Usage:**

```python
from pathlib import Path
from fishinfo.buffer.disk import DiskBuffer

buffer = DiskBuffer(
    db_path=Path("/var/lib/fishinfo/buffer.db"),
    max_size=10000,
    vacuum_interval=1000,
)
await buffer.initialize()

# Same interface as MemoryBuffer
await buffer.push(points)
popped = await buffer.pop(100)

# Close when done
await buffer.close()
```

### 5. Compression (`compression.py`)

gzip compression utilities:
- Configurable compression level
- Automatic threshold detection
- Compression statistics

**Functions:**
- `compress(data, level=6)` - Compress bytes
- `decompress(data)` - Decompress bytes
- `is_compressed(data)` - Check if compressed
- `maybe_compress(data, config)` - Conditionally compress
- `maybe_decompress(data)` - Auto-detect and decompress
- `get_compression_stats(original_size, compressed_size)` - Calculate stats

**Usage:**

```python
from fishinfo.compression import CompressionConfig, maybe_compress

config = CompressionConfig(
    enabled=True,
    threshold=1024,  # Only compress if > 1KB
    level=6,         # Compression level 1-9
)

# Conditionally compress
compressed, was_compressed = maybe_compress(data, config)

# Use with BatchIngestionClient
batch_client = BatchIngestionClient(
    client=client,
    compression=True,
    compression_config=config,
)
```

## Retry Logic

Built-in exponential backoff retry:

```python
from fishinfo.retry import RetryConfig

retry_config = RetryConfig(
    max_retries=3,
    initial_delay=1.0,
    max_delay=60.0,
    exponential_base=2.0,
    jitter=True,
)

batch_client = BatchIngestionClient(
    client=client,
    retry_config=retry_config,
)
```

## Dead Letter Queue

Points that fail after all retries are moved to the DLQ:

```python
# Check DLQ size
dlq_size = batch_client.get_dead_letter_queue_size()

# Get DLQ items
items = await batch_client.get_dead_letter_items()
for item, error in items:
    print(f"Failed: {error}")

# Clear DLQ
await batch_client.clear_dead_letter_queue()
```

## Callbacks

Optional callbacks for monitoring:

```python
def on_batch_sent(count: int, latency_ms: float) -> None:
    print(f"Sent {count} points in {latency_ms:.2f}ms")

def on_batch_failed(count: int, error: str) -> None:
    print(f"Failed to send {count} points: {error}")

batch_client = BatchIngestionClient(
    client=client,
    on_batch_sent=on_batch_sent,
    on_batch_failed=on_batch_failed,
)
```

## Statistics

Track ingestion performance:

```python
stats = batch_client.get_stats()

print(f"Batches sent: {stats.batches_sent}")
print(f"Batches failed: {stats.batches_failed}")
print(f"Points sent: {stats.points_sent}")
print(f"Points failed: {stats.points_failed}")
print(f"Points buffered: {stats.points_buffered}")
print(f"Avg latency: {stats.avg_latency_ms:.2f}ms")
print(f"Min latency: {stats.min_latency_ms:.2f}ms")
print(f"Max latency: {stats.max_latency_ms:.2f}ms")
```

## Testing

Comprehensive test coverage:

```bash
# Run all new tests
pytest tests/test_compression.py -v
pytest tests/test_buffer_memory.py -v
pytest tests/test_buffer_disk.py -v
pytest tests/test_ingestion_batch.py -v

# Run all tests
pytest tests/ -v
```

## Migration Guide

### From BatchingClient (Legacy) to BatchIngestionClient

The new `BatchIngestionClient` is compatible with the legacy `BatchingClient` but provides additional features:

**Before (Legacy):**
```python
from fishinfo.batch import BatchingClient
from fishinfo.buffer import OfflineBuffer

buffer = OfflineBuffer(db_path=Path("buffer.db"))
await buffer.initialize()

batch_client = BatchingClient(
    send_fn=client.send_batch,
    batch_size=100,
    flush_interval=5.0,
    offline_buffer=buffer,
)
await batch_client.start()
```

**After (RFC-040):**
```python
from fishinfo.ingestion import BatchIngestionClient
from fishinfo.buffer.disk import DiskBuffer

buffer = DiskBuffer(db_path=Path("buffer.db"))
await buffer.initialize()

batch_client = BatchIngestionClient(
    client=client,
    max_batch_size=100,
    max_batch_age=5.0,
    buffer=buffer,
    compression=True,
)
await batch_client.start()
```

**Key Differences:**
1. `BatchIngestionClient` takes the full client, not just `send_fn`
2. `batch_size` → `max_batch_size`
3. `flush_interval` → `max_batch_age`
4. `offline_buffer` → `buffer` (accepts any Buffer implementation)
5. Built-in compression support
6. Improved type safety with Buffer protocol

## Performance Considerations

### Memory Usage

**MemoryBuffer:**
- ~1KB per telemetry point (average)
- 10,000 points ≈ 10MB RAM

**DiskBuffer:**
- ~2KB per point on disk (JSON + SQLite overhead)
- 10,000 points ≈ 20MB disk space
- Minimal RAM usage (SQLite caching)

### Compression

Compression ratios (typical):
- JSON telemetry data: 60-80% reduction
- Repetitive position data: 70-85% reduction
- Mixed data types: 50-70% reduction

**Threshold Guidelines:**
- < 1KB: Skip compression (overhead too high)
- 1KB - 10KB: Compress (good balance)
- > 10KB: Always compress (significant savings)

### Batching

Optimal batch sizes:
- **Small vessels (1 sensor):** 10-50 points
- **Medium vessels (5-10 sensors):** 50-100 points
- **Large vessels (20+ sensors):** 100-500 points

Optimal batch age:
- **Real-time monitoring:** 1-2 seconds
- **Normal operations:** 5-10 seconds
- **Offline buffering:** 30-60 seconds

## Error Handling

```python
from fishinfo.exceptions import (
    BufferFullError,
    FishinfoError,
)

try:
    await batch_client.add(point)
except BufferFullError:
    # Buffer is full, cannot add more points
    print("Buffer full, dropping point")
except FishinfoError as e:
    # Other fishinfo errors
    print(f"Error: {e}")
```

## Best Practices

1. **Use context managers** for automatic cleanup:
   ```python
   async with BatchIngestionClient(...) as batch_client:
       # Your code here
       pass
   # Automatically flushed and stopped
   ```

2. **Choose the right buffer**:
   - `MemoryBuffer` for transient data and testing
   - `DiskBuffer` for production with unreliable networks

3. **Enable compression** for production:
   ```python
   config = CompressionConfig(enabled=True, threshold=1024)
   ```

4. **Monitor statistics**:
   ```python
   stats = batch_client.get_stats()
   if stats.points_failed > stats.points_sent * 0.1:
       # More than 10% failure rate
       alert_ops_team()
   ```

5. **Handle DLQ items**:
   ```python
   if batch_client.get_dead_letter_queue_size() > 100:
       # Too many permanently failed points
       items = await batch_client.get_dead_letter_items()
       log_to_monitoring_system(items)
       await batch_client.clear_dead_letter_queue()
   ```

## Backward Compatibility

The legacy `BatchingClient` and `OfflineBuffer` remain available for backward compatibility:

```python
# Legacy imports still work
from fishinfo import BatchingClient, OfflineBuffer

# New imports (RFC-040)
from fishinfo import BatchIngestionClient, MemoryBuffer, DiskBuffer
```

Both can coexist in the same codebase during migration.
