feat: keep RGB-D captures only on workstation

tg3_data_collection/README.md

@@ -2,8 +2,10 @@
 
 机器人只在 `/home/nvidia/tg3_data_collection/ready` 暴露已经收到 SIGINT、写完
 `metadata.yaml`、通过 `ros2 bag info` 并生成 SHA-256 清单的 episode。本机服务用免密 SSH/rsync 复制到
-`Data_Get/.incoming`，逐文件校验后再原子改名为 `Data_Get/<episode_id>`。中断的复制不会
-显示成完成数据，也不会删除机器人上的备份。
+`Data_Get/.incoming`，逐文件校验并 fsync 后再原子改名为 `Data_Get/<episode_id>`。本机
+`VERIFIED` 收据持久化成功后，正式服务会再次完整哈希本机 payload，再让 Nvidia 侧固定根目录
+helper 比对原始 manifest SHA-256，原子移入 `deleting/` 后清理。因此完整数据最终只保留在 PS 本机，Nvidia 只是临时
+staging；中断、哈希错误或本机落盘失败都不会触发远端删除。
 
 数据只走本机到 Nvidia 的 SSH 链路，与公网或 EAI 本地 OmniSocket Hub 无关。当前默认：
 
@@ -12,6 +14,8 @@ nvidia@192.168.41.2:/home/nvidia/tg3_data_collection/ready/
   -> /home/ps/Desktop/TG3_TS1P_OmniSocket_Teleop/Data_Get/
 ```
 
+EAI 不运行 recorder/sync，也不保存 MCAP 或图像数据。
+
 先确认免密与依赖：
 
 ```bash
@@ -20,6 +24,18 @@ command -v rsync
 ssh nvidia@192.168.41.2 command -v rsync
 ```
 
+先把固定根目录删除 helper 部署到 Nvidia。它的 CLI 只接受安全 episode ID 和 64 位
+manifest SHA-256，不接受 root 参数，也不会枚举或触碰 `active/`、`failed/`。helper 在
+`deleting/` 树之外的 `delete_ledger/` 先持久化独立删除授权；即使递归删除中断且 tombstone
+里的 READY/manifest 已消失，仍能凭该授权安全续删，最后才删除授权文件：
+
+```bash
+scp tg3_local_teleop/delete_ready_episode.py \
+  nvidia@192.168.41.2:/home/nvidia/tg3_local_teleop/
+ssh nvidia@192.168.41.2 \
+  chmod 755 /home/nvidia/tg3_local_teleop/delete_ready_episode.py
+```
+
 安装本机用户服务：
 
 ```bash
@@ -39,10 +55,20 @@ python3 -m json.tool Data_Get/sync_status.json
 也可以只执行一次，便于首次部署验收：
 
 ```bash
-python3 tg3_data_collection/data_get_sync.py --once
+systemctl --user stop tg3-data-get-sync.service
+python3 tg3_data_collection/data_get_sync.py --once --delete-remote-after-sync
+systemctl --user start tg3-data-get-sync.service
 ```
 
+同步器使用 `Data_Get/.data_get_sync.lock` 做 flock；正式服务运行时，另一个 `--once` 会明确
+报“已有同步进程”，不会与服务同时复制或删除。不带 `--delete-remote-after-sync` 的手动命令
+只复制、验真并保留远端，正式 unit 已显式启用自动回收。
+
 日常轮询不会每 2 秒重新读取并哈希全部历史 MCAP；首次原子发布时已经完成深度校验。
+如果进程在远端删除 ACK 返回前退出，`VERIFIED` 会保持 `pending`。每次真正调用 helper
+之前都会重新完整哈希本机 payload；失败后从 30 秒开始指数退避，最大 15 分钟，进程重启也
+遵守持久化的 `next_retry_unix_s`，避免每 2 秒重读大型 MCAP。helper 使用独立 600 秒超时，
+能继续清理同摘要 tombstone/ledger，也会把已经不存在的同一目标视为成功。
 需要定期审计已有本地数据时单独执行（可能耗时较长）：
 
 ```bash
@@ -56,6 +82,7 @@ python3 tg3_data_collection/data_get_sync.py --once --verify-existing
 ```text
 <episode_id>/
   READY
+  VERIFIED
   manifest.json
   bag/metadata.yaml
   bag/*.mcap
@@ -64,10 +91,32 @@ python3 tg3_data_collection/data_get_sync.py --once --verify-existing
 ```
 
 同步端会再次核对 `READY`、episode ID、每个 MCAP/metadata 的长度和 SHA-256；远端
-manifest 在传输中发生变化也会拒绝发布最终目录。`.incoming`、`sync_status.json` 和全部
-episode 已由 `Data_Get/.gitignore` 排除。服务绝不自动删除机器人端 `ready/` 数据；确认
-本机备份后如需清理，必须由操作者明确指定具体 episode，不能删除整个项目目录。
+manifest 在传输中发生变化也会拒绝发布最终目录。随后 fsync 所有复制文件和目录、以
+NOREPLACE 语义原子发布、fsync 父目录，最后写入并 fsync `VERIFIED`。收据绑定 episode、
+远端身份、原始 manifest SHA-256、已校验文件数和字节数，并记录 `pending/deleted`。
+`.incoming`、`sync_status.json` 和全部 episode 已由 `Data_Get/.gitignore` 排除。
 
-迁移后需要同步修改 service 中的本机项目路径、机器人 SSH 地址和机器人 ready 路径。
-如果 Nvidia 的 `192.168.41.2` 改变，只改该 unit 的 `--remote`；它不在机器人
-`config.toml` 或 OmniSocket Peer 设置中。
+临时目录按 `<episode>.<manifest_sha256>.partial` 隔离，rsync 同时使用删除同步；发布前还会
+拒绝 manifest 未列出的任何 `*.mcap` 或 `metadata.yaml`，避免旧中断文件混入新 episode。
+
+`sync_status.json` 的 `state=delete_pending` 表示本机数据已完整发布，但远端回收尚未收到
+成功 ACK；`pending_remote_cleanup`、`last_delete_error`、`delete_count` 和
+`last_deleted_episode` 可用于排查。此状态下不要手工删除本机数据，服务会在下一轮重试。
+
+已有本机同名目录不会直接授权删除：没有可信 `VERIFIED` 时必须重新做完整哈希，并要求
+本机与当前远端 manifest 原始摘要一致，之后才补收据并回收远端。同名碰撞、损坏、符号
+链接、错误 root 或摘要不符都保留远端并报错。
+
+历史 `retained/deleted` 收据可以来自旧机器人 IP，不会阻塞新地址的同步；仍为 `pending` 的
+跨 IP 收据不会静默授权新机器人删除，只会逐项报错且不影响其他新 episode 继续同步。
+
+自动回收 helper 严格只覆盖 `ready/`，录制中的 `active/` 永远不能由 PS 同步器处理，
+也不能对 `failed/` 使用 READY 凭据。当前正式配置为
+`retain_failed_episodes=false`：录制器只把失败原因写入状态和 journal，随后精确删除自己
+刚创建的失败 payload，因此机器人不会长期保留失败 MCAP。若以后改回 `true` 以保留故障
+诊断，必须另建独立验真与回收协议，不能整目录清空。
+
+迁移后需要同步修改 service 中的本机项目路径和机器人 SSH 地址。如果 Nvidia 的
+`192.168.41.2` 改变，只改 unit 的 `--remote` 并重新部署 helper；它不在机器人
+`config.toml` 或 OmniSocket Peer 设置中。启用自动删除时远端路径必须保持固定的
+`/home/nvidia/tg3_data_collection/ready`，同步器会拒绝对其他 root 启用删除。

tg3_data_collection/test_data_get_sync.py

@@ -3,12 +3,20 @@ from __future__ import annotations
 
 import hashlib
 import json
+import os
+import shutil
+import subprocess
 import tempfile
 import unittest
 from pathlib import Path
+from unittest import mock
 
 from data_get_sync import (
     DataGetSync,
+    FIXED_REMOTE_READY,
+    ManifestDocument,
+    VERIFIED_RECEIPT_NAME,
+    fsync_episode_tree,
     parse_manifest,
     safe_episode_name,
     safe_relative_path,
@@ -16,6 +24,99 @@ from data_get_sync import (
 )
 
 
+class LocalRsyncRunner:
+    def __init__(self, remote_root: Path) -> None:
+        self.remote_root = remote_root
+        self.mode = "normal"
+        self.rsync_calls = 0
+
+    def run(
+        self, command: list[str], *, timeout: float
+    ) -> subprocess.CompletedProcess[str]:
+        del timeout
+        if not command or command[0] != "rsync":
+            return subprocess.CompletedProcess(command, 1, "", "unexpected command")
+        self.rsync_calls += 1
+        remote_source = command[-2].rstrip("/")
+        episode = remote_source.rsplit("/", 1)[-1]
+        destination = Path(command[-1])
+        shutil.copytree(
+            self.remote_root / episode,
+            destination,
+            dirs_exist_ok=True,
+            symlinks=True,
+        )
+        if self.mode == "truncate":
+            (destination / "bag" / "bag_0.mcap").write_bytes(b"truncated")
+        elif self.mode == "manifest_changed":
+            manifest = destination / "manifest.json"
+            manifest.write_bytes(manifest.read_bytes() + b"\n")
+        elif self.mode == "manifest_symlink":
+            manifest = destination / "manifest.json"
+            manifest.unlink()
+            manifest.symlink_to("bag/metadata.yaml")
+        return subprocess.CompletedProcess(command, 0, "", "")
+
+
+class LocalSync(DataGetSync):
+    def __init__(
+        self,
+        *,
+        remote_root: Path,
+        destination: Path,
+        runner: LocalRsyncRunner,
+        delete: bool = True,
+        remote: str = "nvidia@test",
+    ) -> None:
+        super().__init__(
+            remote=remote,
+            remote_ready=FIXED_REMOTE_READY,
+            destination=destination,
+            status_file=destination / "sync_status.json",
+            runner=runner,
+            reserve_bytes=0,
+            delete_remote_after_sync=delete,
+            remote_delete_helper="/fixed/delete_ready_episode.py",
+        )
+        self.remote_root = remote_root
+        self.delete_calls: list[tuple[str, str]] = []
+        self.delete_outcomes: list[str] = []
+
+    def list_remote_episodes(self) -> list[str]:
+        return sorted(
+            path.name
+            for path in self.remote_root.iterdir()
+            if path.is_dir() and not path.is_symlink()
+        )
+
+    def get_remote_manifest(self, episode: str) -> ManifestDocument:
+        raw = (self.remote_root / episode / "manifest.json").read_bytes()
+        return ManifestDocument.from_bytes(raw, episode)
+
+    def _invoke_delete_helper(
+        self, episode: str, manifest_sha256: str
+    ) -> dict[str, object]:
+        self.delete_calls.append((episode, manifest_sha256))
+        outcome = self.delete_outcomes.pop(0) if self.delete_outcomes else "success"
+        target = self.remote_root / episode
+        if outcome == "failure":
+            raise RuntimeError("injected delete failure")
+        if outcome == "ack_lost_after_delete":
+            if target.exists():
+                shutil.rmtree(target)
+            raise RuntimeError("injected lost acknowledgement")
+        if target.exists():
+            shutil.rmtree(target)
+            state = "deleted"
+        else:
+            state = "already_absent"
+        return {
+            "state": state,
+            "episode_id": episode,
+            "manifest_sha256": manifest_sha256,
+        }
+
+
 class DataGetSyncTests(unittest.TestCase):
     def test_safe_episode_name(self) -> None:
         self.assertTrue(safe_episode_name("episode_20260810T120000000_deadbeef"))
@@ -93,28 +194,377 @@ class DataGetSyncTests(unittest.TestCase):
             with self.assertRaises(ValueError):
                 validate_episode_dir(episode, manifest)
 
-    def test_existing_episode_skips_expensive_hash_unless_requested(self) -> None:
+    def test_fsync_episode_tree_rejects_intermediate_symlink(self) -> None:
         with tempfile.TemporaryDirectory() as temporary:
             root = Path(temporary)
-            name = "episode_20260810T120000000_deadbeef"
-            episode, _manifest = self._episode(root, name)
-            (episode / "bag" / "bag_0.mcap").write_bytes(b"changed")
-            fast = DataGetSync(
-                remote="unused",
-                remote_ready="/unused",
-                destination=root,
-                status_file=root / "status.json",
-            )
-            self.assertFalse(fast.sync_episode(name))
-            deep = DataGetSync(
-                remote="unused",
-                remote_ready="/unused",
-                destination=root,
-                status_file=root / "status.json",
-                verify_existing=True,
-            )
+            episode = root / "episode"
+            episode.mkdir()
+            outside = root / "outside"
+            outside.mkdir()
+            (outside / "file").write_bytes(b"sentinel")
+            (episode / "linked").symlink_to(outside, target_is_directory=True)
             with self.assertRaises(ValueError):
-                deep.sync_episode(name)
+                fsync_episode_tree(episode)
+
+    def _local_sync(
+        self, root: Path, name: str = "episode_20260810T120000000_deadbeef"
+    ) -> tuple[LocalSync, LocalRsyncRunner, Path, Path]:
+        remote = root / "remote_ready"
+        destination = root / "Data_Get"
+        remote.mkdir()
+        destination.mkdir()
+        self._episode(remote, name)
+        runner = LocalRsyncRunner(remote)
+        syncer = LocalSync(
+            remote_root=remote,
+            destination=destination,
+            runner=runner,
+        )
+        return syncer, runner, remote, destination
+
+    def test_copy_is_verified_receipted_then_remote_is_deleted(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary:
+            name = "episode_20260810T120000000_deadbeef"
+            syncer, _runner, remote, destination = self._local_sync(
+                Path(temporary), name
+            )
+            self.assertEqual(syncer.run_once(), 1)
+            final = destination / name
+            receipt = json.loads((final / VERIFIED_RECEIPT_NAME).read_text())
+            self.assertEqual(receipt["state"], "VERIFIED")
+            self.assertEqual(receipt["remote_identity"], syncer.remote_identity)
+            self.assertEqual(receipt["verified_files"], 2)
+            self.assertGreater(receipt["verified_bytes"], 0)
+            self.assertEqual(receipt["remote_delete"]["state"], "deleted")
+            self.assertFalse((remote / name).exists())
+            self.assertEqual(syncer.delete_count, 1)
+            status = json.loads((destination / "sync_status.json").read_text())
+            self.assertEqual(status["state"], "idle")
+            self.assertEqual(status["pending_remote_cleanup"], 0)
+
+    def test_truncated_copy_never_calls_delete(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary:
+            syncer, runner, remote, _destination = self._local_sync(Path(temporary))
+            runner.mode = "truncate"
+            with self.assertRaises(RuntimeError):
+                syncer.run_once()
+            self.assertEqual(syncer.delete_calls, [])
+            self.assertEqual(len(list(remote.iterdir())), 1)
+
+    def test_manifest_change_during_copy_never_calls_delete(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary:
+            syncer, runner, remote, _destination = self._local_sync(Path(temporary))
+            runner.mode = "manifest_changed"
+            with self.assertRaises(RuntimeError):
+                syncer.run_once()
+            self.assertEqual(syncer.delete_calls, [])
+            self.assertEqual(len(list(remote.iterdir())), 1)
+
+    def test_copied_manifest_symlink_never_calls_delete(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary:
+            syncer, runner, remote, _destination = self._local_sync(Path(temporary))
+            runner.mode = "manifest_symlink"
+            with self.assertRaises(RuntimeError):
+                syncer.run_once()
+            self.assertEqual(syncer.delete_calls, [])
+            self.assertEqual(len(list(remote.iterdir())), 1)
+
+    def test_delete_failure_is_pending_and_retried(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary:
+            name = "episode_20260810T120000000_deadbeef"
+            syncer, _runner, remote, destination = self._local_sync(
+                Path(temporary), name
+            )
+            syncer.delete_outcomes = ["failure", "success"]
+            self.assertEqual(syncer.run_once(), 1)
+            receipt_path = destination / name / VERIFIED_RECEIPT_NAME
+            receipt = json.loads(receipt_path.read_text())
+            self.assertEqual(receipt["remote_delete"]["state"], "pending")
+            self.assertTrue((remote / name).exists())
+            status = json.loads((destination / "sync_status.json").read_text())
+            self.assertEqual(status["state"], "delete_pending")
+            self.assertEqual(status["pending_remote_cleanup"], 1)
+
+            # Persistent backoff prevents a 2-second poll loop from repeatedly
+            # hashing a large MCAP after a network/helper failure.
+            self.assertEqual(syncer.run_once(), 0)
+            self.assertEqual(len(syncer.delete_calls), 1)
+            receipt = json.loads(receipt_path.read_text())
+            receipt["remote_delete"]["next_retry_unix_s"] = 0.0
+            receipt_path.write_text(json.dumps(receipt), encoding="utf-8")
+
+            self.assertEqual(syncer.run_once(), 0)
+            receipt = json.loads(receipt_path.read_text())
+            self.assertEqual(receipt["remote_delete"]["state"], "deleted")
+            self.assertFalse((remote / name).exists())
+            self.assertEqual(len(syncer.delete_calls), 2)
+
+    def test_lost_delete_ack_is_idempotent_after_process_restart(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary:
+            name = "episode_20260810T120000000_deadbeef"
+            root = Path(temporary)
+            first, runner, remote, destination = self._local_sync(root, name)
+            first.delete_outcomes = ["ack_lost_after_delete"]
+            self.assertEqual(first.run_once(), 1)
+            self.assertFalse((remote / name).exists())
+            receipt_path = destination / name / VERIFIED_RECEIPT_NAME
+            self.assertEqual(
+                json.loads(receipt_path.read_text())["remote_delete"]["state"],
+                "pending",
+            )
+
+            restarted = LocalSync(
+                remote_root=remote,
+                destination=destination,
+                runner=runner,
+            )
+            with mock.patch(
+                "data_get_sync.validate_episode_dir",
+                wraps=validate_episode_dir,
+            ) as deep_validator:
+                self.assertEqual(restarted.run_once(), 0)
+                self.assertEqual(deep_validator.call_count, 0)
+                self.assertEqual(restarted.delete_calls, [])
+
+                receipt = json.loads(receipt_path.read_text())
+                receipt["remote_delete"]["next_retry_unix_s"] = 0.0
+                receipt_path.write_text(json.dumps(receipt), encoding="utf-8")
+                self.assertEqual(restarted.run_once(), 0)
+                self.assertGreaterEqual(deep_validator.call_count, 1)
+            self.assertEqual(
+                json.loads(receipt_path.read_text())["remote_delete"]["state"],
+                "deleted",
+            )
+            self.assertEqual(restarted.delete_calls[0][0], name)
+
+    def test_retry_rehashes_and_refuses_delete_after_local_tamper(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary:
+            name = "episode_20260810T120000000_deadbeef"
+            syncer, _runner, remote, destination = self._local_sync(
+                Path(temporary), name
+            )
+            syncer.delete_outcomes = ["failure", "success"]
+            self.assertEqual(syncer.run_once(), 1)
+            (destination / name / "bag" / "bag_0.mcap").write_bytes(b"tampered")
+            receipt_path = destination / name / VERIFIED_RECEIPT_NAME
+            receipt = json.loads(receipt_path.read_text())
+            receipt["remote_delete"]["next_retry_unix_s"] = 0.0
+            receipt_path.write_text(json.dumps(receipt), encoding="utf-8")
+
+            with self.assertRaises(RuntimeError):
+                syncer.run_once()
+            self.assertEqual(len(syncer.delete_calls), 1)
+            self.assertTrue((remote / name).exists())
+
+    def test_unlisted_stale_mcap_is_never_published_or_deleted(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary:
+            name = "episode_20260810T120000000_deadbeef"
+            syncer, _runner, remote, destination = self._local_sync(
+                Path(temporary), name
+            )
+            (remote / name / "bag" / "stale_old_payload.mcap").write_bytes(
+                b"stale"
+            )
+            with self.assertRaises(RuntimeError):
+                syncer.run_once()
+            self.assertFalse((destination / name).exists())
+            self.assertEqual(syncer.delete_calls, [])
+            self.assertTrue((remote / name).exists())
+
+    def test_reused_hash_staging_cannot_smuggle_old_mcap(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary:
+            name = "episode_20260810T120000000_deadbeef"
+            syncer, _runner, remote, destination = self._local_sync(
+                Path(temporary), name
+            )
+            document = syncer.get_remote_manifest(name)
+            staging = (
+                destination
+                / ".incoming"
+                / f"{name}.{document.sha256}.partial"
+                / "bag"
+            )
+            staging.mkdir(parents=True)
+            (staging / "stale_old_payload.mcap").write_bytes(b"old interrupted bag")
+            with self.assertRaises(RuntimeError):
+                syncer.run_once()
+            self.assertFalse((destination / name).exists())
+            self.assertEqual(syncer.delete_calls, [])
+            self.assertTrue((remote / name).exists())
+
+    def test_deleted_receipt_from_old_ip_does_not_block_new_ip(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary:
+            name = "episode_20260810T120000000_deadbeef"
+            root = Path(temporary)
+            first, runner, remote, destination = self._local_sync(root, name)
+            self.assertEqual(first.run_once(), 1)
+            receipt_path = destination / name / VERIFIED_RECEIPT_NAME
+            receipt = json.loads(receipt_path.read_text())
+            receipt["remote"] = "nvidia@old-ip"
+            receipt["remote_identity"] = (
+                f"nvidia@old-ip:{FIXED_REMOTE_READY}"
+            )
+            receipt_path.write_text(json.dumps(receipt), encoding="utf-8")
+
+            migrated = LocalSync(
+                remote_root=remote,
+                destination=destination,
+                runner=runner,
+                remote="nvidia@new-ip",
+            )
+            self.assertEqual(migrated.run_once(), 0)
+            status = json.loads((destination / "sync_status.json").read_text())
+            self.assertEqual(status["state"], "idle")
+            self.assertEqual(status["pending_remote_cleanup"], 0)
+
+    def test_bad_pending_old_ip_does_not_starve_new_episode(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary:
+            root = Path(temporary)
+            old_name = "episode_old"
+            first, _runner, _old_remote, destination = self._local_sync(
+                root, old_name
+            )
+            first.delete_outcomes = ["failure"]
+            self.assertEqual(first.run_once(), 1)
+
+            new_remote = root / "new_remote_ready"
+            new_remote.mkdir()
+            new_name = "episode_new"
+            self._episode(new_remote, new_name)
+            new_runner = LocalRsyncRunner(new_remote)
+            migrated = LocalSync(
+                remote_root=new_remote,
+                destination=destination,
+                runner=new_runner,
+                remote="nvidia@new-ip",
+            )
+            self.assertEqual(migrated.run_once(), 1)
+            self.assertFalse((new_remote / new_name).exists())
+            self.assertTrue((destination / new_name).is_dir())
+            status = json.loads((destination / "sync_status.json").read_text())
+            self.assertEqual(status["state"], "delete_pending")
+            self.assertEqual(status["pending_remote_cleanup"], 1)
+            self.assertIn("remote_identity", status["last_delete_error"])
+
+    def test_delete_helper_uses_independent_long_timeout(self) -> None:
+        class AckRunner:
+            def __init__(self) -> None:
+                self.timeouts: list[float] = []
+
+            def run(
+                self, command: list[str], *, timeout: float
+            ) -> subprocess.CompletedProcess[str]:
+                self.timeouts.append(timeout)
+                payload = {
+                    "state": "already_absent",
+                    "episode_id": "episode_timeout",
+                    "manifest_sha256": "0" * 64,
+                }
+                return subprocess.CompletedProcess(
+                    command, 0, json.dumps(payload) + "\n", ""
+                )
+
+        with tempfile.TemporaryDirectory() as temporary:
+            runner = AckRunner()
+            syncer = DataGetSync(
+                remote="nvidia@test",
+                remote_ready=FIXED_REMOTE_READY,
+                destination=Path(temporary),
+                status_file=Path(temporary) / "status.json",
+                runner=runner,
+                delete_remote_after_sync=True,
+                remote_delete_timeout_s=600.0,
+            )
+            syncer._invoke_delete_helper("episode_timeout", "0" * 64)
+            self.assertEqual(runner.timeouts, [600.0])
+
+    def test_existing_final_without_receipt_is_deep_verified_before_delete(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary:
+            name = "episode_20260810T120000000_deadbeef"
+            root = Path(temporary)
+            syncer, _runner, remote, destination = self._local_sync(root, name)
+            shutil.copytree(remote / name, destination / name)
+            with mock.patch(
+                "data_get_sync.validate_episode_dir",
+                wraps=validate_episode_dir,
+            ) as deep_validator:
+                self.assertEqual(syncer.run_once(), 0)
+                self.assertGreaterEqual(deep_validator.call_count, 1)
+            self.assertTrue((destination / name / VERIFIED_RECEIPT_NAME).is_file())
+            self.assertFalse((remote / name).exists())
+
+    def test_existing_corrupt_final_without_receipt_never_deletes_remote(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary:
+            name = "episode_20260810T120000000_deadbeef"
+            root = Path(temporary)
+            syncer, _runner, remote, destination = self._local_sync(root, name)
+            shutil.copytree(remote / name, destination / name)
+            (destination / name / "bag" / "bag_0.mcap").write_bytes(b"corrupt")
+            with self.assertRaises(RuntimeError):
+                syncer.run_once()
+            self.assertEqual(syncer.delete_calls, [])
+            self.assertTrue((remote / name).exists())
+
+    def test_existing_manifest_collision_never_deletes_remote(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary:
+            name = "episode_20260810T120000000_deadbeef"
+            root = Path(temporary)
+            syncer, _runner, remote, destination = self._local_sync(root, name)
+            self._episode(destination, name)
+            local_manifest = destination / name / "manifest.json"
+            local_manifest.write_bytes(local_manifest.read_bytes() + b"\n")
+            with self.assertRaises(RuntimeError):
+                syncer.run_once()
+            self.assertEqual(syncer.delete_calls, [])
+            self.assertTrue((remote / name).exists())
+
+    def test_receipt_write_failure_never_calls_delete(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary:
+            syncer, _runner, remote, _destination = self._local_sync(Path(temporary))
+            with mock.patch(
+                "data_get_sync.atomic_write_json",
+                side_effect=RuntimeError("injected receipt fsync failure"),
+            ):
+                with self.assertRaises(RuntimeError):
+                    syncer.sync_episode(next(remote.iterdir()).name)
+            self.assertEqual(syncer.delete_calls, [])
+            self.assertEqual(len(list(remote.iterdir())), 1)
+
+    def test_sync_flock_rejects_concurrent_once(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary:
+            root = Path(temporary)
+            first, runner, remote, destination = self._local_sync(root)
+            second = LocalSync(
+                remote_root=remote,
+                destination=destination,
+                runner=runner,
+            )
+            with first.process_lock():
+                with self.assertRaisesRegex(RuntimeError, "already holds"):
+                    second.run_once()
+
+    def test_symlink_lock_is_refused(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary:
+            root = Path(temporary)
+            syncer, _runner, _remote, destination = self._local_sync(root)
+            sentinel = root / "sentinel"
+            sentinel.write_text("do not touch", encoding="utf-8")
+            (destination / ".data_get_sync.lock").symlink_to(sentinel)
+            with self.assertRaises(RuntimeError):
+                syncer.run_once()
+            self.assertEqual(sentinel.read_text(encoding="utf-8"), "do not touch")
+
+    def test_delete_mode_rejects_nonfixed_remote_ready(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary:
+            root = Path(temporary)
+            with self.assertRaises(ValueError):
+                DataGetSync(
+                    remote="nvidia@test",
+                    remote_ready="/tmp/not-ready",
+                    destination=root,
+                    status_file=root / "status.json",
+                    delete_remote_after_sync=True,
+                )
 
     def test_bad_old_episode_does_not_starve_newer_episode(self) -> None:
         with tempfile.TemporaryDirectory() as temporary:

tg3_data_collection/tg3-data-get-sync.service

@@ -4,7 +4,7 @@ Description=TG3 completed data episode sync to Data_Get
 [Service]
 Type=simple
 WorkingDirectory=/home/ps/Desktop/TG3_TS1P_OmniSocket_Teleop
-ExecStart=/usr/bin/python3 /home/ps/Desktop/TG3_TS1P_OmniSocket_Teleop/tg3_data_collection/data_get_sync.py --remote nvidia@192.168.41.2 --remote-ready /home/nvidia/tg3_data_collection/ready --destination /home/ps/Desktop/TG3_TS1P_OmniSocket_Teleop/Data_Get --poll-seconds 2
+ExecStart=/usr/bin/python3 /home/ps/Desktop/TG3_TS1P_OmniSocket_Teleop/tg3_data_collection/data_get_sync.py --remote nvidia@192.168.41.2 --remote-ready /home/nvidia/tg3_data_collection/ready --destination /home/ps/Desktop/TG3_TS1P_OmniSocket_Teleop/Data_Get --delete-remote-after-sync --poll-seconds 2
 Restart=always
 RestartSec=2
 KillSignal=SIGINT