rseng-big-data-processing

Covers processing research data that outgrows one machine's memory: out-of-core and chunked computation, Dask for scaling the scientific Python stack, Spark for distributed tabular pipelines, lazy evaluation, partitioning strategies, idempotent and restartable batch jobs, and knowing when NOT to distribute. Use when datasets no longer fit in memory, when the user mentions Dask, Spark, out-of-core or larger-than-memory data, when a pandas/NumPy workflow hits memory limits, or when designing batch pipelines over many files. (Cluster job submission and job arrays are rseng-hpc-computing; pipeline orchestration engines are rseng-workflows; profile first with rseng-performance-profiling.)

存档时间线

版本索引永久保留；内容副本只保留最新 1 份，更早版本仅留索引与哈希（存档时间线的证据链不会因此断裂）。

下载存档内容副本