EthoPipe Deterministic ETL Pipeline
Decoupling semantic unstructured ingestion from strict biological serialization into Darwin Core relational schemas. Eliminating anthropomorphic variance in canine behavioral epidemiology.
Four-Stage Decoupled Data Pipeline
Ingestion Gate
FastAPI Stream Gateway
FastAPI application entrypoint at src.pipeline.main:app. Consumes raw UTF-8 streams, shelter incident notes, and uncalibrated veterinary intake logs.
Semantic Extraction
Clamped Zero-Temp Clerk
gemini-3.5-flash locked strictly to T = 0.0. Deterministically extracts kinematic actions and strips moralizing prose via forced JSON Schema.
Pydantic Guardrail
Veterinary Biological Lock
Pydantic v2 ConfigDict(strict=True, extra='forbid'). Enforces Merck Veterinary bounds (30–250 BPM); invalid anomalies are quarantined.
DwC Serialization
Global Biodiversity Standard
Outputs Kimball Star Schema, Parquet files, and DwC-A records with segregated HumanObservation vs MachineObservation.
Live Schema Verification & Transformation Matrix
from datetime import datetime
from pydantic import BaseModel, ConfigDict, Field, field_validator
from typing import Literal, Optional
class CanineObservationRecord(BaseModel):
model_config = ConfigDict(strict=True, extra="forbid")
individual_id: str = Field(..., pattern=r"^CANIS-[A-Z0-9]+$")
event_timestamp: datetime
ethogram_token: Literal[
"play_bow",
"displacement_oral_lip_licking",
"displacement_pant",
"pinnae_caudal_retraction",
"locomotion_avoidance_vector"
]
heart_rate_bpm: Optional[int] = Field(None, ge=30, le=250)
basis_of_record: Literal["MachineObservation", "HumanObservation"]
@field_validator("heart_rate_bpm")
def validate_merck_canine_bounds(cls, value: Optional[int]) -> Optional[int]:
if value is not None and not (30 <= value <= 250):
raise ValueError(
"Physiological anomaly: BPM violates Merck canine boundaries (30-250 BPM)."
)
return value[
{
"dwc:basisOfRecord": "HumanObservation",
"dwc:individualID": "CANIS-SUBJ-8821",
"dwc:eventDate": "2026-07-05T12:00:00Z",
"dwc:measurementType": "displacement_oral_mastication",
"dwc:measurementValue": "1"
},
{
"dwc:basisOfRecord": "HumanObservation",
"dwc:individualID": "CANIS-SUBJ-8821",
"dwc:eventDate": "2026-07-05T12:00:00Z",
"dwc:measurementType": "displacement_oral_lip_licking",
"dwc:measurementValue": "4"
},
{
"dwc:basisOfRecord": "MachineObservation",
"dwc:individualID": "CANIS-SUBJ-8821",
"dwc:eventDate": "2026-07-05T12:00:00Z",
"dwc:measurementType": "heart_rate_bpm",
"dwc:measurementValue": "95"
}
]Multi-Stage CI Matrix Guardrails
Black Formatter
Strict 88-character column line limits and AST-level uncompromising style uniformity across all modules.
Ruff Linter
High-velocity static syntax anomaly and anti-pattern detection executing on every pull request.
Pytest Multi-Matrix
Isolated container sandbox validation across both Python 3.11 and 3.12 runtime kernels simultaneously.
Darwin Core Schema Gate
Automated validation against Darwin Core data dictionaries prior to branch merges, preventing ontological drift.
Deterministic Local Execution (uv First)
Environment Lock
Clone the canonical repository and synchronize virtual environment dependencies strictly with uv.lock.
git clone https://github.com/sothiss/ethopipe.git
cd ethopipe && uv sync
FastAPI Application
Spins up the continuous biotelemetry ingestion API (src.pipeline.main:app) on port 8000.
uv run uvicorn src.pipeline.main:app --reload --port 8000
Automated QA Pipeline
Executes Black, Ruff, the 43-test passing Pytest multi-matrix, and the Solo Maintainer Guardrail auditor.
.\scripts\run_qa.ps1
43 PASS
Grounding Biological Literature
EthoPipe operational definitions are directly grounded in published behavioral ethograms and peer-reviewed veterinary benchmarks:
Broseghini et al. (2024). Ethogram of the predatory sequence of dogs (Canis familiaris). Applied Animal Behaviour Science, 279, 106402.
de Winkel et al. (2024). Observational behaviors and emotions to assess welfare of dogs: A systematic review. Journal of Veterinary Behavior, 72, 1–17.
Normal Physiological Values for Dogs: Baseline resting canine heart rates (60–140 BPM), extreme biological telemetry boundaries (30–250 BPM).
Institutional Allocations
Subsidized compute allocations supporting high-throughput token serialization:
Integrate EthoPipe with Your Research Telemetry
Clone the deterministic parser, ingest unformatted incident notes, or connect your continuous IMU/ECG streams to the Darwin Core serialization sink.