Mirror the archive.
Bulk downloads run as data-transfer jobs into cluster scratch. We record the source, version, access date, size and record count.
04 / APPENDIX
How we know
what we claim.
Every chart in this project comes from public data we downloaded, processed on MIT's Engaging cluster and analysed with committed code. This page lists the datasets, the jobs, the methods and the limits behind each result.
Jump to the analyses ↓192 GBpublic data processed
256Mrecords analysed
36distinct public datasets
64Slurm jobs · 15 CPU-hours
Totals count each dataset and each cluster job once, even when several analyses use them. "Processed" bytes were downloaded or streamed and parsed. Separately, we catalogued 30.3 TB of NASA's ISS physical-science archive object by object; that listing is not counted as processed.
04.0 / THE BRIEF, LINE BY LINE
“Real space data goes in.”
Here is what went in.
The SpaceXAI challenge asks for enormous public datasets, decades of missions and a literature no human can read all of. Each line of the brief maps to an analysis below.
04.1 / THE PIPELINE
Archive in.
Evidence out.
The same four steps for every analysis. Raw data stays on the cluster; the site renders compact result files that anyone can regenerate.
Shard and parallelise.
Slurm job arrays parse archives in independent shards and write columnar tables. Every job ID, partition and runtime is kept with the result.
Count the right units.
A protein, a run or a satellite is the unit of evidence, not a frame or a row. Matched comparisons, bootstrap intervals and stated confounders come before any headline.
Small, honest files.
Each analysis writes one JSON result: question, headline, charts, findings, limitations, next steps and the commands to reproduce it. The site reads those files directly.
04.2 / EVIDENCE LEVELS
Every number
has a label.
| LABEL | MEANING | WHAT IT CAN SUPPORT |
|---|---|---|
| Published result | A value taken from a peer-reviewed paper or agency document, with the page or table cited. | That someone measured it, under their conditions. |
| Our computation on public data | We downloaded the data and computed the value with our code. | A pattern in the archive. Not, by itself, a causal effect of microgravity. |
| Physics computation | A calculation from standard physical laws with stated inputs. | Orders of magnitude and design bounds, not measurements. |
| Simulation | A seeded model with stated assumptions. | When a policy helps or hurts inside the model. Not flight performance. |
| Hypothesis | Something we intend to test. | A reason to run an experiment. |
04.3 / EVERY ANALYSIS
Data, compute,
method and limits.
Grouped by chapter. Open "Reproduce & code" for the exact commands and source files.
Loading analyses…
04.4 / TOOLS & INFRASTRUCTURE
What we built with.
| TOOL | ROLE IN THIS PROJECT |
|---|---|
| MIT ORCD Engaging | Slurm job arrays on mit_normal, bulk downloads on mit_data_transfer, and 1 TB of flash scratch for raw archives. Job IDs are listed with each analysis. |
| Python data stack | Streaming parsers, Parquet tables and NumPy/pandas statistics. The libraries for each analysis are in its methods note and requirements. |
| Static site | Native ES modules, SVG and canvas. No runtime CDN, no tracking. Charts render from the committed result files. |
| Version control | Code, result files and methods notes live in the project repository; raw archives stay on the cluster. |
AI coding and research assistants were used under our direction to write analysis code and draft text. Every number on this site regenerates from the committed code and public data; nothing in a result file is hand-edited.
04.5 / REPRODUCE IT
Run it yourself.
git clone https://github.com/fvrarturo/spacexai-calhacks-13 cd spacexai-calhacks-13 npm run analyze # published arithmetic + fleet benchmark npm test # analysis invariants # big-data analyses: see docs/appendix/*.md for each workstream's Slurm jobs npm run build # regenerates the site from result files
Earlier work is preserved: the research desk, the original concept story, the results report and the methods notes.