Disk Space
For users interested in downloading AbacusSummit data, this breakdown of the file sizes of the various data products may be helpful.
The following is an annotated summary of du -BMB (so units of 1,000,000 bytes, as disk drives use).
This is for AbacusSummit_base_c000_ph008, which is typical, supplemented
by AbacusSummit_base_c000_ph001 for some information about the full time slices.
# The total outputs are 7.85 TB per simulation, if there are no full time slices.
7851475 AbacusSummit_base_c000_ph008
# When there are full time slices, they are about 3.5 TB *each*,
# with a mild trend in redshift, due to slight changes in compression efficiency.
# Sims have between 0 and 12 full time-slice epochs.
3578826 slices/z0.200
3534315 slices/z0.800
3498498 slices/z1.400
3472193 slices/z2.000
3451782 slices/z2.500
3431805 slices/z3.000
# Here's the breakdown; most of the data is in the halos.
# Most users won't need to look at the logs.
1 [top-level directory]
1 info
5649 log
1274121 lightcones
6571688 halos
# The lightcones contain 10% of the particles in rv+pid, and 100% in heal.
# rv: particle pos/vel
# pid: can be used to connect particles in the lightcone to halos in the timeslices;
# also contains the kernel density estimate from the fully sampled density field.
# heal: The Nside=16384, ring=True healpix projection
# These are broken into files for each time step, typically d(ln(1+z)) ~ 0.002.
703830 lightcones/rv
329456 lightcones/heal
240836 lightcones/pid
# The halos come in two flavors: 12 primary time slices with more outputs,
# and then 21 secondary time slices with only limited outputs.
# Most users will only want the primary slices; the secondary ones
# are intended for merger trees and for matching halos onto the light cones.
#
# The primary slices are bigger, typically 400 GB each:
412661 halos/z0.100
412863 halos/z0.200
412748 halos/z0.300
412340 halos/z0.400
411654 halos/z0.500
408158 halos/z0.800
402861 halos/z1.100
396109 halos/z1.400
388176 halos/z1.700
379538 halos/z2.000
364212 halos/z2.500
348863 halos/z3.000
# Total of primary slices: 4750 GB
# The secondary slices are smaller, typically 100 GB and decreasing at high z:
111493 halos/z0.150
110879 halos/z0.250
110041 halos/z0.350
109006 halos/z0.450
107437 halos/z0.575
106324 halos/z0.650
105147 halos/z0.725
102498 halos/z0.875
101105 halos/z0.950
99575 halos/z1.025
96330 halos/z1.175
94619 halos/z1.250
92831 halos/z1.325
89070 halos/z1.475
87182 halos/z1.550
85311 halos/z1.625
79098 halos/z1.850
67995 halos/z2.250
53962 halos/z2.750
11267 halos/z5.000
351 halos/z8.000
# Total of secondary slices: 1822 GB
# But one might opt only to keep certain types of files on disk, so here is the summary
# of the types.
# For example, a minimal installation might be only the halo_info and halo_rv_A files,
# which are 1.1 TB, and perhaps only for some of the primary epochs. E.g., five
# epochs might knock the storage down to 0.5 TB per simulation.
# Another example would be to install only the primary epochs, without the
# field B samples. This saves 1.83 TB per sim, so one is at 6 TB/sim.
# The halo info files contain the stats about all halos, typically 70-75 GB/epoch.
# Note that the file format supports reading only subsets of columns; many users
# will need to load only a small fraction of these.
73188 halos/z0.100 halo_info
74301 halos/z0.200 halo_info
75190 halos/z0.300 halo_info
75864 halos/z0.400 halo_info
76323 halos/z0.500 halo_info
76416 halos/z0.800 halo_info
74722 halos/z1.100 halo_info
71488 halos/z1.400 halo_info
67014 halos/z1.700 halo_info
61615 halos/z2.000 halo_info
51419 halos/z2.500 halo_info
40797 halos/z3.000 halo_info
# Primary halo_info: 818 GB
73764 halos/z0.150 halo_info
74763 halos/z0.250 halo_info
75550 halos/z0.350 halo_info
76107 halos/z0.450 halo_info
76508 halos/z0.575 halo_info
76591 halos/z0.650 halo_info
76548 halos/z0.725 halo_info
76120 halos/z0.875 halo_info
75755 halos/z0.950 halo_info
75270 halos/z1.025 halo_info
74008 halos/z1.175 halo_info
73240 halos/z1.250 halo_info
72376 halos/z1.325 halo_info
70387 halos/z1.475 halo_info
69312 halos/z1.550 halo_info
68204 halos/z1.625 halo_info
64279 halos/z1.850 halo_info
56553 halos/z2.250 halo_info
45891 halos/z2.750 halo_info
10066 halos/z5.000 halo_info
303 halos/z8.000 halo_info
# Secondary halo_info: 1362 GB
# The particles associated to these halos, with 3% consistent subsample
# in A and 7% in B, all indexed out of the halo_info files.
# First, we have the positions and velocities.
# Users painting HOD satellite galaxies into the halos probably could just use the A set.
31415 halos/z0.100 halo_rv_A
30859 halos/z0.200 halo_rv_A
30213 halos/z0.300 halo_rv_A
29498 halos/z0.400 halo_rv_A
28733 halos/z0.500 halo_rv_A
26264 halos/z0.800 halo_rv_A
23699 halos/z1.100 halo_rv_A
21216 halos/z1.400 halo_rv_A
18767 halos/z1.700 halo_rv_A
16457 halos/z2.000 halo_rv_A
12965 halos/z2.500 halo_rv_A
9965 halos/z3.000 halo_rv_A
# Primary 3% halo rv: 280 GB
70448 halos/z0.100 halo_rv_B
69199 halos/z0.200 halo_rv_B
67735 halos/z0.300 halo_rv_B
66123 halos/z0.400 halo_rv_B
64399 halos/z0.500 halo_rv_B
58825 halos/z0.800 halo_rv_B
53036 halos/z1.100 halo_rv_B
47321 halos/z1.400 halo_rv_B
41824 halos/z1.700 halo_rv_B
36626 halos/z2.000 halo_rv_B
28821 halos/z2.500 halo_rv_B
22132 halos/z3.000 halo_rv_B
# Primary 7% halo rv: 626 GB
# And then we have the PIDs, with kernel densities embedded.
# These are used build merger trees, but could also be used
# to track particular particles as part of galaxy assignment,
# e.g., to find the densest particle in a progenitor halo and
# use its late-time position.
13680 halos/z0.100 halo_pid_A
13325 halos/z0.200 halo_pid_A
12947 halos/z0.300 halo_pid_A
12548 halos/z0.400 halo_pid_A
12140 halos/z0.500 halo_pid_A
10870 halos/z0.800 halo_pid_A
9652 halos/z1.100 halo_pid_A
8493 halos/z1.400 halo_pid_A
7421 halos/z1.700 halo_pid_A
6442 halos/z2.000 halo_pid_A
4989 halos/z2.500 halo_pid_A
3777 halos/z3.000 halo_pid_A
# Primary 3% halo pid: 116 GB
30217 halos/z0.100 halo_pid_B
29390 halos/z0.200 halo_pid_B
28501 halos/z0.300 halo_pid_B
27584 halos/z0.400 halo_pid_B
26637 halos/z0.500 halo_pid_B
23780 halos/z0.800 halo_pid_B
21020 halos/z1.100 halo_pid_B
18439 halos/z1.400 halo_pid_B
16061 halos/z1.700 halo_pid_B
13899 halos/z2.000 halo_pid_B
10731 halos/z2.500 halo_pid_B
8097 halos/z3.000 halo_pid_B
# Primary 7% halo pid: 254 GB
# For the secondary epochs, we provide only the PID+density file.
# These are slightly smaller because they include only L1 particles, not L0 particles.
11726 halos/z0.150 halo_pid_A
11238 halos/z0.250 halo_pid_A
10743 halos/z0.350 halo_pid_A
10256 halos/z0.450 halo_pid_A
9655 halos/z0.575 halo_pid_A
9292 halos/z0.650 halo_pid_A
8940 halos/z0.725 halo_pid_A
8258 halos/z0.875 halo_pid_A
7941 halos/z0.950 halo_pid_A
7620 halos/z1.025 halo_pid_A
7007 halos/z1.175 halo_pid_A
6716 halos/z1.250 halo_pid_A
6429 halos/z1.325 halo_pid_A
5883 halos/z1.475 halo_pid_A
5628 halos/z1.550 halo_pid_A
5389 halos/z1.625 halo_pid_A
4680 halos/z1.850 halo_pid_A
3621 halos/z2.250 halo_pid_A
2563 halos/z2.750 halo_pid_A
395 halos/z5.000 halo_pid_A
16 halos/z8.000 halo_pid_A
# Secondary 3% halo pid: 144 GB
26003 halos/z0.150 halo_pid_B
24879 halos/z0.250 halo_pid_B
23749 halos/z0.350 halo_pid_B
22643 halos/z0.450 halo_pid_B
21274 halos/z0.575 halo_pid_B
20442 halos/z0.650 halo_pid_B
19659 halos/z0.725 halo_pid_B
18120 halos/z0.875 halo_pid_B
17410 halos/z0.950 halo_pid_B
16685 halos/z1.025 halo_pid_B
15316 halos/z1.175 halo_pid_B
14664 halos/z1.250 halo_pid_B
14026 halos/z1.325 halo_pid_B
12802 halos/z1.475 halo_pid_B
12243 halos/z1.550 halo_pid_B
11719 halos/z1.625 halo_pid_B
10139 halos/z1.850 halo_pid_B
7822 halos/z2.250 halo_pid_B
5509 halos/z2.750 halo_pid_B
806 halos/z5.000 halo_pid_B
34 halos/z8.000 halo_pid_B
# Secondary 7% halo pid: 316 GB
# We provide the rest of the density field, i.e., the complement of the halo set,
# in the subsamples. These would be used in matter-field statistics or if one wanted
# to associate particles in the periphery of halos. Or if one wanted to run a different
# group finder (admittedly on only 10% of the dynamical particles).
43432 halos/z0.100 field_rv_A
44044 halos/z0.200 field_rv_A
44724 halos/z0.300 field_rv_A
45445 halos/z0.400 field_rv_A
46195 halos/z0.500 field_rv_A
48542 halos/z0.800 field_rv_A
50898 halos/z1.100 field_rv_A
53164 halos/z1.400 field_rv_A
55295 halos/z1.700 field_rv_A
57277 halos/z2.000 field_rv_A
60189 halos/z2.500 field_rv_A
62610 halos/z3.000 field_rv_A
# Primary 3% field rv: 612 GB
97497 halos/z0.100 field_rv_B
98866 halos/z0.200 field_rv_B
100381 halos/z0.300 field_rv_B
101991 halos/z0.400 field_rv_B
103666 halos/z0.500 field_rv_B
108916 halos/z0.800 field_rv_B
114209 halos/z1.100 field_rv_B
119330 halos/z1.400 field_rv_B
124160 halos/z1.700 field_rv_B
128634 halos/z2.000 field_rv_B
135233 halos/z2.500 field_rv_B
140699 halos/z3.000 field_rv_B
# Primary 7% field rv: 1374 GB
# Field PIDs are probably not used much, but these do relate particles across epochs
# and the PID encodes the initial grid location for Lagrangian displacements.
16508 halos/z0.100 field_pid_A
16556 halos/z0.200 field_pid_A
16627 halos/z0.300 field_pid_A
16709 halos/z0.400 field_pid_A
16800 halos/z0.500 field_pid_A
17124 halos/z0.800 field_pid_A
17468 halos/z1.100 field_pid_A
17809 halos/z1.400 field_pid_A
18128 halos/z1.700 field_pid_A
18443 halos/z2.000 field_pid_A
18833 halos/z2.500 field_pid_A
19093 halos/z3.000 field_pid_A
# Primary 3% field pid: 210 GB
36280 halos/z0.100 field_pid_B
36327 halos/z0.200 field_pid_B
36434 halos/z0.300 field_pid_B
36582 halos/z0.400 field_pid_B
36765 halos/z0.500 field_pid_B
37426 halos/z0.800 field_pid_B
38160 halos/z1.100 field_pid_B
38855 halos/z1.400 field_pid_B
39510 halos/z1.700 field_pid_B
40150 halos/z2.000 field_pid_B
41037 halos/z2.500 field_pid_B
41698 halos/z3.000 field_pid_B
# Primary 7% field pid: 459 GB
# For the full time slices, they are split into L0 and field (non-L0) sets.
# However, this was just due to convenience in the code; the L0 particles
# are not indexed in halo_info. Only concatenations will be useful.
# The fractional split of L0 to field increases to low redshift.
#
# The position+velocity data is in the pack9 format, which gives
# somewhat higher precision than RVint. These files average about
# 2.8 TB per epoch, which is 8.5 bytes per particle.
1682068 slices/z0.200/field_pack9
1083440 slices/z0.200/L0_pack9
1864771 slices/z0.800/field_pack9
925909 slices/z0.800/L0_pack9
2061023 slices/z1.400/field_pack9
748046 slices/z1.400/L0_pack9
2242314 slices/z2.000/field_pack9
581440 slices/z2.000/L0_pack9
2371831 slices/z2.500/field_pack9
458272 slices/z2.500/L0_pack9
2480799 slices/z3.000/field_pack9
352132 slices/z3.000/L0_pack9
# The PID and kernel density estimate are in the pid files.
# These average about 0.7 TB per epoch, increasing toward low redshift.
510862 slices/z3.000/field_pack9_pid
88014 slices/z3.000/L0_pack9_pid
501868 slices/z2.500/field_pack9_pid
119812 slices/z2.500/L0_pack9_pid
489859 slices/z2.000/field_pack9_pid
158581 slices/z2.000/L0_pack9_pid
472529 slices/z1.400/field_pack9_pid
216902 slices/z1.400/L0_pack9_pid
454954 slices/z0.800/field_pack9_pid
288682 slices/z0.800/L0_pack9_pid
444629 slices/z0.200/field_pack9_pid
368691 slices/z0.200/L0_pack9_pid