[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"summary-1c37c1cad77c687a-scale-pytorch-ddp-multi-node-on-aws-ec2-infra-firs-summary":3,"summaries-facets-categories":252,"summary-related-1c37c1cad77c687a-scale-pytorch-ddp-multi-node-on-aws-ec2-infra-firs-summary":3837},{"id":4,"title":5,"ai":6,"body":13,"categories":216,"created_at":218,"date_modified":218,"description":48,"extension":219,"faq":218,"featured":220,"kicker_label":218,"meta":221,"navigation":103,"path":235,"published_at":236,"question":218,"scraped_at":237,"seo":238,"sitemap":239,"source_id":240,"source_name":241,"source_type":242,"source_url":243,"stem":244,"tags":245,"thumbnail_url":218,"tldr":249,"tweet":218,"unknown_tags":250,"__hash__":251},"summaries\u002Fsummaries\u002F1c37c1cad77c687a-scale-pytorch-ddp-multi-node-on-aws-ec2-infra-firs-summary.md","Scale PyTorch DDP Multi-Node on AWS EC2: Infra-First Guide",{"provider":7,"model":8,"input_tokens":9,"output_tokens":10,"processing_time_ms":11,"cost_usd":12},"openrouter","x-ai\u002Fgrok-4.1-fast",8453,1898,16685,0.0026171,{"type":14,"value":15,"toc":210},"minimark",[16,21,25,28,32,35,38,53,56,60,63,189,196,199,203,206],[17,18,20],"h2",{"id":19},"replicate-environments-and-data-for-multi-node-reliability","Replicate Environments and Data for Multi-Node Reliability",[22,23,24],"p",{},"Multi-node DDP treats processes across independent EC2 instances as identical, requiring each node to have matching Python\u002FPyTorch\u002FCUDA versions, identical code from version control, and shared dataset access. Use shared EFS volumes mounted on all instances (e.g., DATASET_DIR=\u002Fefs\u002Fandrea\u002Fdataset) to avoid copying data; local copies or remote streaming work but add latency. Homogeneous clusters like 2 g6e.xlarge instances in the same availability zone minimize variance. Without this, expect cryptic errors or silent failures since DDP assumes uniformity.",[22,26,27],{},"One process per GPU (world size = total GPUs, e.g., 2 for 1 GPU\u002Fnode), with rank 0 as master for logging\u002Fcheckpointing. NCCL handles intra-node (NVLink\u002FPCIe) and inter-node (TCP) gradient all-reduce; network misconfigs cause silent hangs.",[17,29,31],{"id":30},"secure-aws-networking-and-launch-torchrun","Secure AWS Networking and Launch torchrun",[22,33,34],{},"Launch identical instance types, note master's private IP (e.g., 10.x.xxx.203), and edit security group inbound rules: Type=All traffic, Source=same security group ID (e.g., sg-xxx). This enables rendezvous and NCCL comms; default blocks cause indefinite hangs without errors.",[22,36,37],{},"Set .env per node:",[39,40,41,50],"ul",{},[42,43,44,45],"li",{},"Master: NUMBER_OF_NODES=2, NODE_RANK=0, NUMBER_OF_GPUS=1, MASTER_ADDR=",[46,47,49],"private",{"ip":48},"",", MASTER_PORT=30000, DDP_TIMEOUT_SECONDS=180",[42,51,52],{},"Worker: Same but NODE_RANK=1, OUTPUT_DIR empty (master-only).",[22,54,55],{},"Run in tmux: uv run torchrun --nnodes=2 --node_rank=$NODE_RANK --nproc_per_node=1 --master_addr=$MASTER_ADDR --master_port=30000 train.py. Batch size scales linearly (e.g., per-rank batch_size=10 yields effective 20), adjust LR accordingly.",[17,57,59],{"id":58},"integrate-ddpmanager-and-distributedsampler-in-code","Integrate DDPManager and DistributedSampler in Code",[22,61,62],{},"Encapsulate DDP in DDPManager class:",[64,65,69],"pre",{"className":66,"code":67,"language":68,"meta":48,"style":48},"language-python shiki shiki-themes github-light github-dark","import os\nimport torch\nimport torch.distributed as dist\nfrom datetime import timedelta\n\nclass DDPManager:\n    def __init__(self, backend=\"nccl\", timeout_s=180):\n        self.backend = backend\n        self.timeout_s = timeout_s\n    def setup(self) -> bool:\n        if dist.is_initialized(): return True\n        if \"RANK\" not in os.environ: return False\n        local_rank = int(os.environ[\"LOCAL_RANK\"])\n        torch.cuda.set_device(local_rank)\n        dist.init_process_group(backend=self.backend, timeout=timedelta(seconds=self.timeout_s))\n        return True\n    def is_main_process(self) -> bool:\n        return int(os.environ.get(\"RANK\", \"0\")) == 0\n    # barrier(), cleanup(), get_local_rank()\n","python",[70,71,72,80,86,92,98,105,111,117,123,129,135,141,147,153,159,165,171,177,183],"code",{"__ignoreMap":48},[73,74,77],"span",{"class":75,"line":76},"line",1,[73,78,79],{},"import os\n",[73,81,83],{"class":75,"line":82},2,[73,84,85],{},"import torch\n",[73,87,89],{"class":75,"line":88},3,[73,90,91],{},"import torch.distributed as dist\n",[73,93,95],{"class":75,"line":94},4,[73,96,97],{},"from datetime import timedelta\n",[73,99,101],{"class":75,"line":100},5,[73,102,104],{"emptyLinePlaceholder":103},true,"\n",[73,106,108],{"class":75,"line":107},6,[73,109,110],{},"class DDPManager:\n",[73,112,114],{"class":75,"line":113},7,[73,115,116],{},"    def __init__(self, backend=\"nccl\", timeout_s=180):\n",[73,118,120],{"class":75,"line":119},8,[73,121,122],{},"        self.backend = backend\n",[73,124,126],{"class":75,"line":125},9,[73,127,128],{},"        self.timeout_s = timeout_s\n",[73,130,132],{"class":75,"line":131},10,[73,133,134],{},"    def setup(self) -> bool:\n",[73,136,138],{"class":75,"line":137},11,[73,139,140],{},"        if dist.is_initialized(): return True\n",[73,142,144],{"class":75,"line":143},12,[73,145,146],{},"        if \"RANK\" not in os.environ: return False\n",[73,148,150],{"class":75,"line":149},13,[73,151,152],{},"        local_rank = int(os.environ[\"LOCAL_RANK\"])\n",[73,154,156],{"class":75,"line":155},14,[73,157,158],{},"        torch.cuda.set_device(local_rank)\n",[73,160,162],{"class":75,"line":161},15,[73,163,164],{},"        dist.init_process_group(backend=self.backend, timeout=timedelta(seconds=self.timeout_s))\n",[73,166,168],{"class":75,"line":167},16,[73,169,170],{},"        return True\n",[73,172,174],{"class":75,"line":173},17,[73,175,176],{},"    def is_main_process(self) -> bool:\n",[73,178,180],{"class":75,"line":179},18,[73,181,182],{},"        return int(os.environ.get(\"RANK\", \"0\")) == 0\n",[73,184,186],{"class":75,"line":185},19,[73,187,188],{},"    # barrier(), cleanup(), get_local_rank()\n",[22,190,191,192,195],{},"Setup: ddp = DDPManager(); use_ddp = ddp.setup(); device = torch.device(f\"cuda:{ddp.get_local_rank()}\") if use_ddp else \"cuda:0\". Wrap model: model = DDP(model, device_ids=",[73,193,194],{},"local_rank",", output_device=local_rank, find_unused_parameters=False); access via model.module.",[22,197,198],{},"Use DistributedSampler(dataset, num_replicas=world_size, rank=rank, shuffle=True) for data partitioning; set train_sampler.set_epoch(epoch) per epoch. Barrier after master-only tasks (validate\u002Fsave): if use_ddp: ddp.barrier(). Master handles checkpoints: torch.save({\"step\": step, \"model\": model.module.state_dict()}, f\"{ckpt_dir}\u002Fmodel-{step}.pth\").",[17,200,202],{"id":201},"debug-timeouts-and-failures-proactively","Debug Timeouts and Failures Proactively",[22,204,205],{},"Silent hangs signal network issues—ping test instances first. Missing node triggers init timeout (180s default). Master crash kills job; no fault tolerance. Deadlocks (e.g., barrier stall) timeout. Restrict GPUs: export CUDA_VISIBLE_DEVICES=0. Scale batch size with ranks for stable training; effective batch = per-rank batch * world_size.",[207,208,209],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":48,"searchDepth":82,"depth":82,"links":211},[212,213,214,215],{"id":19,"depth":82,"text":20},{"id":30,"depth":82,"text":31},{"id":58,"depth":82,"text":59},{"id":201,"depth":82,"text":202},[217],"DevOps & Cloud",null,"md",false,{"content_references":222,"triage":232},[223,228],{"type":224,"title":225,"url":226,"context":227},"other","Mounting the EFS file system on EC2 Linux","https:\u002F\u002Fdocs.aws.amazon.com\u002Fefs\u002Flatest\u002Fug\u002Fmounting-fs-mount-helper-ec2-linux.html","mentioned",{"type":229,"title":230,"url":231,"context":227},"tool","tmux","https:\u002F\u002Fman7.org\u002Flinux\u002Fman-pages\u002Fman1\u002Ftmux.1.html",{"relevance":100,"novelty":88,"quality":94,"actionability":94,"composite":233,"reasoning":234},4.15,"Category: AI & LLMs. The article provides a detailed guide on scaling PyTorch DDP across AWS EC2 instances, addressing practical challenges faced by developers in deploying AI models. It includes specific configurations and code examples that can be directly applied, making it actionable for the target audience.","\u002Fsummaries\u002F1c37c1cad77c687a-scale-pytorch-ddp-multi-node-on-aws-ec2-infra-firs-summary","2026-04-30 13:31:01","2026-05-03 17:01:04",{"title":5,"description":48},{"loc":235},"1c37c1cad77c687a","Learning Data","article","https:\u002F\u002Fmedium.com\u002Flearning-data\u002Fone-gpu-wasnt-enough-my-journey-scaling-pytorch-ddp-across-aws-ec2-instances-506647e086fc?source=rss----eec44e936bf1---4","summaries\u002F1c37c1cad77c687a-scale-pytorch-ddp-multi-node-on-aws-ec2-infra-firs-summary",[68,246,247,248],"machine-learning","devops","cloud","Multi-node DDP demands identical environments, data access, and open security groups across EC2 instances; use torchrun launcher with DDPManager for minimal code changes and reliable gradient sync via NCCL.",[],"IVwD5gQ2TAKP9L1byc-qt16swkQ8B55VMXVdjNlKNQ0",[253,256,259,262,265,268,270,272,274,276,278,280,283,285,287,289,291,293,295,297,299,301,304,307,309,311,314,316,318,321,323,325,327,329,331,333,335,337,339,341,343,345,347,349,351,353,355,357,359,361,363,365,367,369,371,373,375,377,379,381,383,385,387,389,391,393,395,397,399,401,403,405,407,409,411,413,415,417,419,421,423,425,427,429,431,433,435,437,439,441,443,445,447,449,451,453,455,457,459,461,463,465,467,469,471,473,475,477,479,481,483,485,487,489,491,493,495,497,499,501,503,505,507,509,511,513,515,517,519,521,523,525,527,529,531,533,535,537,539,541,543,545,547,549,551,553,555,557,559,561,563,565,567,569,571,573,575,577,579,581,583,585,587,589,591,593,595,597,599,601,603,605,607,609,611,613,615,617,619,621,623,625,627,629,631,633,635,637,639,641,643,645,647,649,651,653,655,657,659,661,663,665,667,669,671,673,675,677,679,681,683,685,687,689,691,693,695,697,699,701,703,705,707,709,711,713,715,717,719,721,723,725,727,729,731,733,735,737,739,741,743,745,747,749,751,753,755,757,759,761,763,765,767,769,771,773,775,777,779,781,783,785,787,789,791,793,795,797,799,801,803,805,807,809,811,813,815,817,819,821,823,825,827,829,831,833,835,837,839,841,843,845,847,849,851,853,855,857,859,861,863,865,867,869,871,873,875,877,879,881,883,885,887,889,891,893,895,897,899,901,903,905,907,909,911,913,915,917,919,921,923,925,927,929,931,933,935,937,939,941,943,945,947,949,951,953,955,957,959,961,963,965,967,969,971,973,975,977,979,981,983,985,987,989,991,993,995,997,999,1001,1003,1005,1007,1009,1011,1013,1015,1017,1019,1021,1023,1025,1027,1029,1031,1033,1035,1037,1039,1041,1043,1045,1047,1049,1051,1053,1055,1057,1059,1061,1063,1065,1067,1069,1071,1073,1075,1077,1079,1081,1083,1085,1087,1089,1091,1093,1095,1097,1099,1101,1103,1105,1107,1109,1111,1113,1115,1117,1119,1121,1123,1125,1127,1129,1131,1133,1135,1137,1139,1141,1143,1145,1147,1149,1151,1153,1155,1157,1159,1161,1163,1165,1167,1169,1171,1173,1175,1177,1179,1181,1183,1185,1187,1189,1191,1193,1195,1197,1199,1201,1203,1205,1207,1209,1211,1213,1215,1217,1219,1221,1223,1225,1227,1229,1231,1233,1235,1237,1239,1241,1243,1245,1247,1249,1251,1253,1255,1257,1259,1261,1263,1265,1267,1269,1271,1273,1275,1277,1279,1281,1283,1285,1287,1289,1291,1293,1295,1297,1299,1301,1303,1305,1307,1309,1311,1313,1315,1317,1319,1321,1323,1325,1327,1329,1331,1333,1335,1337,1339,1341,1343,1345,1347,1349,1351,1353,1355,1357,1359,1361,1363,1365,1367,1369,1371,1373,1375,1377,1379,1381,1383,1385,1387,1389,1391,1393,1395,1397,1399,1401,1403,1405,1407,1409,1411,1413,1415,1417,1419,1421,1423,1425,1427,1429,1431,1433,1435,1437,1439,1441,1443,1445,1447,1449,1451,1453,1455,1457,1459,1461,1463,1465,1467,1469,1471,1473,1475,1477,1479,1481,1483,1485,1487,1489,1491,1493,1495,1497,1499,1501,1503,1505,1507,1509,1511,1513,1515,1517,1519,1521,1523,1525,1527,1529,1531,1533,1535,1537,1539,1541,1543,1545,1547,1549,1551,1553,1555,1557,1559,1561,1563,1565,1567,1569,1571,1573,1575,1577,1579,1581,1583,1585,1587,1589,1591,1593,1595,1597,1599,1601,1603,1605,1607,1609,1611,1613,1615,1617,1619,1621,1623,1625,1627,1629,1631,1633,1635,1637,1639,1641,1643,1645,1647,1649,1651,1653,1655,1657,1659,1661,1663,1665,1667,1669,1671,1673,1675,1677,1679,1681,1683,1685,1687,1689,1691,1693,1695,1697,1699,1701,1703,1705,1707,1709,1711,1713,1715,1717,1719,1721,1723,1725,1727,1729,1731,1733,1735,1737,1739,1741,1743,1745,1747,1749,1751,1753,1755,1757,1759,1761,1763,1765,1767,1769,1771,1773,1775,1777,1779,1781,1783,1785,1787,1789,1791,1793,1795,1797,1799,1801,1803,1805,1807,1809,1811,1813,1815,1817,1819,1821,1823,1825,1827,1829,1831,1833,1835,1837,1839,1841,1843,1845,1847,1849,1851,1853,1855,1857,1859,1861,1863,1865,1867,1869,1871,1873,1875,1877,1879,1881,1883,1885,1887,1889,1891,1893,1895,1897,1899,1901,1903,1905,1907,1909,1911,1913,1915,1917,1919,1921,1923,1925,1927,1929,1931,1933,1935,1937,1939,1941,1943,1945,1947,1949,1951,1953,1955,1957,1959,1961,1963,1965,1967,1969,1971,1973,1975,1977,1979,1981,1983,1985,1987,1989,1991,1993,1995,1997,1999,2001,2003,2005,2007,2009,2011,2013,2015,2017,2019,2021,2023,2025,2027,2029,2031,2033,2035,2037,2039,2041,2043,2045,2047,2049,2051,2053,2055,2057,2059,2061,2063,2065,2067,2069,2071,2073,2075,2077,2079,2081,2083,2085,2087,2089,2091,2093,2095,2097,2099,2101,2103,2105,2107,2109,2111,2113,2115,2117,2119,2121,2123,2125,2127,2129,2131,2133,2135,2137,2139,2141,2143,2145,2147,2149,2151,2153,2155,2157,2159,2161,2163,2165,2167,2169,2171,2173,2175,2177,2179,2181,2183,2185,2187,2189,2191,2193,2195,2197,2199,2201,2203,2205,2207,2209,2211,2213,2215,2217,2219,2221,2223,2225,2227,2229,2231,2233,2235,2237,2239,2241,2243,2245,2247,2249,2251,2253,2255,2257,2259,2261,2263,2265,2267,2269,2271,2273,2275,2277,2279,2281,2283,2285,2287,2289,2291,2293,2295,2297,2299,2301,2303,2305,2307,2309,2311,2313,2315,2317,2319,2321,2323,2325,2327,2329,2331,2333,2335,2337,2339,2341,2343,2345,2347,2349,2351,2353,2355,2357,2359,2361,2363,2365,2367,2369,2371,2373,2375,2377,2379,2381,2383,2385,2387,2389,2391,2393,2395,2397,2399,2401,2403,2405,2407,2409,2411,2413,2415,2417,2419,2421,2423,2425,2427,2429,2431,2433,2435,2437,2439,2441,2443,2445,2447,2449,2451,2453,2455,2457,2459,2461,2463,2465,2467,2469,2471,2473,2475,2477,2479,2481,2483,2485,2487,2489,2491,2493,2495,2497,2499,2501,2503,2505,2507,2509,2511,2513,2515,2517,2519,2521,2523,2525,2527,2529,2531,2533,2535,2537,2539,2541,2543,2545,2547,2549,2551,2553,2555,2557,2559,2561,2563,2565,2567,2569,2571,2573,2575,2577,2579,2581,2583,2585,2587,2589,2591,2593,2595,2597,2599,2601,2603,2605,2607,2609,2611,2613,2615,2617,2619,2621,2623,2625,2627,2629,2631,2633,2635,2637,2639,2641,2643,2645,2647,2649,2651,2653,2655,2657,2659,2661,2663,2665,2667,2669,2671,2673,2675,2677,2679,2681,2683,2685,2687,2689,2691,2693,2695,2697,2699,2701,2703,2705,2707,2709,2711,2713,2715,2717,2719,2721,2723,2725,2727,2729,2731,2733,2735,2737,2739,2741,2743,2745,2747,2749,2751,2753,2755,2757,2759,2761,2763,2765,2767,2769,2771,2773,2775,2777,2779,2781,2783,2785,2787,2789,2791,2793,2795,2797,2799,2801,2803,2805,2807,2809,2811,2813,2815,2817,2819,2821,2823,2825,2827,2829,2831,2833,2835,2837,2839,2841,2843,2845,2847,2849,2851,2853,2855,2857,2859,2861,2863,2865,2867,2869,2871,2873,2875,2877,2879,2881,2883,2885,2887,2889,2891,2893,2895,2897,2899,2901,2903,2905,2907,2909,2911,2913,2915,2917,2919,2921,2923,2925,2927,2929,2931,2933,2935,2937,2939,2941,2943,2945,2947,2949,2951,2953,2955,2957,2959,2961,2963,2965,2967,2969,2971,2973,2975,2977,2979,2981,2983,2985,2987,2989,2991,2993,2995,2997,2999,3001,3003,3005,3007,3009,3011,3013,3015,3017,3019,3021,3023,3025,3027,3029,3031,3033,3035,3037,3039,3041,3043,3045,3047,3049,3051,3053,3055,3057,3059,3061,3063,3065,3067,3069,3071,3073,3075,3077,3079,3081,3083,3085,3087,3089,3091,3093,3095,3097,3099,3101,3103,3105,3107,3109,3111,3113,3115,3117,3119,3121,3123,3125,3127,3129,3131,3133,3135,3137,3139,3141,3143,3145,3147,3149,3151,3153,3155,3157,3159,3161,3163,3165,3167,3169,3171,3173,3175,3177,3179,3181,3183,3185,3187,3189,3191,3193,3195,3197,3199,3201,3203,3205,3207,3209,3211,3213,3215,3217,3219,3221,3223,3225,3227,3229,3231,3233,3235,3237,3239,3241,3243,3245,3247,3249,3251,3253,3255,3257,3259,3261,3263,3265,3267,3269,3271,3273,3275,3277,3279,3281,3283,3285,3287,3289,3291,3293,3295,3297,3299,3301,3303,3305,3307,3309,3311,3313,3315,3317,3319,3321,3323,3325,3327,3329,3331,3333,3335,3337,3339,3341,3343,3345,3347,3349,3351,3353,3355,3357,3359,3361,3363,3365,3367,3369,3371,3373,3375,3377,3379,3381,3383,3385,3387,3389,3391,3393,3395,3397,3399,3401,3403,3405,3407,3409,3411,3413,3415,3417,3419,3421,3423,3425,3427,3429,3431,3433,3435,3437,3439,3441,3443,3445,3447,3449,3451,3453,3455,3457,3459,3461,3463,3465,3467,3469,3471,3473,3475,3477,3479,3481,3483,3485,3487,3489,3491,3493,3495,3497,3499,3501,3503,3505,3507,3509,3511,3513,3515,3517,3519,3521,3523,3525,3527,3529,3531,3533,3535,3537,3539,3541,3543,3545,3547,3549,3551,3553,3555,3557,3559,3561,3563,3565,3567,3569,3571,3573,3575,3577,3579,3581,3583,3585,3587,3589,3591,3593,3595,3597,3599,3601,3603,3605,3607,3609,3611,3613,3615,3617,3619,3621,3623,3625,3627,3629,3631,3633,3635,3637,3639,3641,3643,3645,3647,3649,3651,3653,3655,3657,3659,3661,3663,3665,3667,3669,3671,3673,3675,3677,3679,3681,3683,3685,3687,3689,3691,3693,3695,3697,3699,3701,3703,3705,3707,3709,3711,3713,3715,3717,3719,3721,3723,3725,3727,3729,3731,3733,3735,3737,3739,3741,3743,3745,3747,3749,3751,3753,3755,3757,3759,3761,3763,3765,3767,3769,3771,3773,3775,3777,3779,3781,3783,3785,3787,3789,3791,3793,3795,3797,3799,3801,3803,3805,3807,3809,3811,3813,3815,3817,3819,3821,3823,3825,3827,3829,3831,3833,3835],{"categories":254},[255],"Developer Productivity",{"categories":257},[258],"Business & SaaS",{"categories":260},[261],"AI & LLMs",{"categories":263},[264],"AI Automation",{"categories":266},[267],"Product Strategy",{"categories":269},[261],{"categories":271},[255],{"categories":273},[258],{"categories":275},[],{"categories":277},[261],{"categories":279},[],{"categories":281},[282],"AI News & Trends",{"categories":284},[264],{"categories":286},[282],{"categories":288},[264],{"categories":290},[264],{"categories":292},[261],{"categories":294},[261],{"categories":296},[282],{"categories":298},[261],{"categories":300},[],{"categories":302},[303],"Design & Frontend",{"categories":305},[306],"Data Science & Visualization",{"categories":308},[282],{"categories":310},[],{"categories":312},[313],"Software Engineering",{"categories":315},[261],{"categories":317},[264],{"categories":319},[320],"Marketing & Growth",{"categories":322},[261],{"categories":324},[264],{"categories":326},[],{"categories":328},[],{"categories":330},[303],{"categories":332},[264],{"categories":334},[255],{"categories":336},[303],{"categories":338},[261],{"categories":340},[264],{"categories":342},[282],{"categories":344},[],{"categories":346},[],{"categories":348},[264],{"categories":350},[313],{"categories":352},[],{"categories":354},[258],{"categories":356},[],{"categories":358},[],{"categories":360},[264],{"categories":362},[264],{"categories":364},[261],{"categories":366},[],{"categories":368},[313],{"categories":370},[],{"categories":372},[],{"categories":374},[],{"categories":376},[261],{"categories":378},[320],{"categories":380},[303],{"categories":382},[303],{"categories":384},[261],{"categories":386},[264],{"categories":388},[261],{"categories":390},[261],{"categories":392},[264],{"categories":394},[264],{"categories":396},[306],{"categories":398},[282],{"categories":400},[264],{"categories":402},[320],{"categories":404},[264],{"categories":406},[267],{"categories":408},[],{"categories":410},[264],{"categories":412},[],{"categories":414},[264],{"categories":416},[313],{"categories":418},[303],{"categories":420},[261],{"categories":422},[],{"categories":424},[],{"categories":426},[264],{"categories":428},[],{"categories":430},[261],{"categories":432},[],{"categories":434},[255],{"categories":436},[313],{"categories":438},[258],{"categories":440},[282],{"categories":442},[261],{"categories":444},[],{"categories":446},[261],{"categories":448},[],{"categories":450},[313],{"categories":452},[306],{"categories":454},[],{"categories":456},[261],{"categories":458},[303],{"categories":460},[],{"categories":462},[303],{"categories":464},[264],{"categories":466},[],{"categories":468},[264],{"categories":470},[282],{"categories":472},[258],{"categories":474},[261],{"categories":476},[],{"categories":478},[264],{"categories":480},[261],{"categories":482},[267],{"categories":484},[],{"categories":486},[261],{"categories":488},[264],{"categories":490},[264],{"categories":492},[],{"categories":494},[306],{"categories":496},[261],{"categories":498},[],{"categories":500},[255],{"categories":502},[258],{"categories":504},[261],{"categories":506},[264],{"categories":508},[313],{"categories":510},[261],{"categories":512},[],{"categories":514},[],{"categories":516},[261],{"categories":518},[],{"categories":520},[303],{"categories":522},[],{"categories":524},[261],{"categories":526},[],{"categories":528},[264],{"categories":530},[261],{"categories":532},[303],{"categories":534},[],{"categories":536},[261],{"categories":538},[261],{"categories":540},[258],{"categories":542},[264],{"categories":544},[261],{"categories":546},[303],{"categories":548},[264],{"categories":550},[],{"categories":552},[],{"categories":554},[282],{"categories":556},[],{"categories":558},[261],{"categories":560},[258,320],{"categories":562},[],{"categories":564},[261],{"categories":566},[],{"categories":568},[],{"categories":570},[261],{"categories":572},[],{"categories":574},[261],{"categories":576},[217],{"categories":578},[],{"categories":580},[282],{"categories":582},[303],{"categories":584},[],{"categories":586},[282],{"categories":588},[282],{"categories":590},[261],{"categories":592},[320],{"categories":594},[],{"categories":596},[258],{"categories":598},[],{"categories":600},[261,217],{"categories":602},[261],{"categories":604},[261],{"categories":606},[264],{"categories":608},[261,313],{"categories":610},[306],{"categories":612},[261],{"categories":614},[320],{"categories":616},[264],{"categories":618},[264],{"categories":620},[],{"categories":622},[264],{"categories":624},[261,258],{"categories":626},[],{"categories":628},[303],{"categories":630},[303],{"categories":632},[],{"categories":634},[],{"categories":636},[282],{"categories":638},[],{"categories":640},[255],{"categories":642},[313],{"categories":644},[261],{"categories":646},[303],{"categories":648},[264],{"categories":650},[313],{"categories":652},[282],{"categories":654},[303],{"categories":656},[],{"categories":658},[261],{"categories":660},[261],{"categories":662},[261],{"categories":664},[282],{"categories":666},[255],{"categories":668},[261],{"categories":670},[264],{"categories":672},[217],{"categories":674},[303],{"categories":676},[264],{"categories":678},[],{"categories":680},[],{"categories":682},[303],{"categories":684},[282],{"categories":686},[306],{"categories":688},[],{"categories":690},[261],{"categories":692},[261],{"categories":694},[258],{"categories":696},[261],{"categories":698},[261],{"categories":700},[282],{"categories":702},[],{"categories":704},[264],{"categories":706},[313],{"categories":708},[],{"categories":710},[261],{"categories":712},[261],{"categories":714},[264],{"categories":716},[],{"categories":718},[],{"categories":720},[261],{"categories":722},[],{"categories":724},[258],{"categories":726},[264],{"categories":728},[],{"categories":730},[255],{"categories":732},[261],{"categories":734},[258],{"categories":736},[282],{"categories":738},[],{"categories":740},[],{"categories":742},[],{"categories":744},[282],{"categories":746},[282],{"categories":748},[],{"categories":750},[],{"categories":752},[258],{"categories":754},[],{"categories":756},[],{"categories":758},[255],{"categories":760},[],{"categories":762},[320],{"categories":764},[264],{"categories":766},[258],{"categories":768},[264],{"categories":770},[313],{"categories":772},[],{"categories":774},[267],{"categories":776},[303],{"categories":778},[313],{"categories":780},[261],{"categories":782},[264],{"categories":784},[258],{"categories":786},[261],{"categories":788},[],{"categories":790},[],{"categories":792},[313],{"categories":794},[306],{"categories":796},[267],{"categories":798},[264],{"categories":800},[261],{"categories":802},[],{"categories":804},[217],{"categories":806},[],{"categories":808},[264],{"categories":810},[],{"categories":812},[],{"categories":814},[261],{"categories":816},[303],{"categories":818},[320],{"categories":820},[264],{"categories":822},[],{"categories":824},[255],{"categories":826},[],{"categories":828},[282],{"categories":830},[261,217],{"categories":832},[282],{"categories":834},[261],{"categories":836},[258],{"categories":838},[261],{"categories":840},[],{"categories":842},[258],{"categories":844},[],{"categories":846},[313],{"categories":848},[303],{"categories":850},[282],{"categories":852},[306],{"categories":854},[255],{"categories":856},[261],{"categories":858},[313],{"categories":860},[],{"categories":862},[],{"categories":864},[267],{"categories":866},[],{"categories":868},[261],{"categories":870},[],{"categories":872},[303],{"categories":874},[303],{"categories":876},[303],{"categories":878},[],{"categories":880},[],{"categories":882},[282],{"categories":884},[264],{"categories":886},[261],{"categories":888},[261],{"categories":890},[261],{"categories":892},[258],{"categories":894},[261],{"categories":896},[],{"categories":898},[313],{"categories":900},[313],{"categories":902},[258],{"categories":904},[],{"categories":906},[261],{"categories":908},[261],{"categories":910},[258],{"categories":912},[282],{"categories":914},[320],{"categories":916},[264],{"categories":918},[],{"categories":920},[303],{"categories":922},[],{"categories":924},[261],{"categories":926},[],{"categories":928},[258],{"categories":930},[264],{"categories":932},[],{"categories":934},[217],{"categories":936},[306],{"categories":938},[313],{"categories":940},[320],{"categories":942},[313],{"categories":944},[264],{"categories":946},[],{"categories":948},[],{"categories":950},[264],{"categories":952},[255],{"categories":954},[264],{"categories":956},[267],{"categories":958},[258],{"categories":960},[],{"categories":962},[261],{"categories":964},[267],{"categories":966},[261],{"categories":968},[261],{"categories":970},[320],{"categories":972},[303],{"categories":974},[264],{"categories":976},[],{"categories":978},[],{"categories":980},[217],{"categories":982},[313],{"categories":984},[],{"categories":986},[264],{"categories":988},[261],{"categories":990},[303,261],{"categories":992},[255],{"categories":994},[],{"categories":996},[261],{"categories":998},[255],{"categories":1000},[303],{"categories":1002},[264],{"categories":1004},[313],{"categories":1006},[],{"categories":1008},[261],{"categories":1010},[],{"categories":1012},[255],{"categories":1014},[],{"categories":1016},[264],{"categories":1018},[267],{"categories":1020},[261],{"categories":1022},[261],{"categories":1024},[303],{"categories":1026},[264],{"categories":1028},[217],{"categories":1030},[303],{"categories":1032},[264],{"categories":1034},[261],{"categories":1036},[261],{"categories":1038},[261],{"categories":1040},[282],{"categories":1042},[],{"categories":1044},[267],{"categories":1046},[264],{"categories":1048},[303],{"categories":1050},[264],{"categories":1052},[313],{"categories":1054},[303],{"categories":1056},[264],{"categories":1058},[282],{"categories":1060},[],{"categories":1062},[261],{"categories":1064},[303],{"categories":1066},[261],{"categories":1068},[255],{"categories":1070},[282],{"categories":1072},[261],{"categories":1074},[320],{"categories":1076},[261],{"categories":1078},[261],{"categories":1080},[264],{"categories":1082},[264],{"categories":1084},[261],{"categories":1086},[264],{"categories":1088},[303],{"categories":1090},[261],{"categories":1092},[],{"categories":1094},[],{"categories":1096},[313],{"categories":1098},[],{"categories":1100},[255],{"categories":1102},[217],{"categories":1104},[],{"categories":1106},[255],{"categories":1108},[258],{"categories":1110},[320],{"categories":1112},[],{"categories":1114},[258],{"categories":1116},[],{"categories":1118},[],{"categories":1120},[],{"categories":1122},[],{"categories":1124},[],{"categories":1126},[261],{"categories":1128},[264],{"categories":1130},[217],{"categories":1132},[255],{"categories":1134},[261],{"categories":1136},[313],{"categories":1138},[267],{"categories":1140},[261],{"categories":1142},[320],{"categories":1144},[261],{"categories":1146},[261],{"categories":1148},[261],{"categories":1150},[261,255],{"categories":1152},[313],{"categories":1154},[313],{"categories":1156},[303],{"categories":1158},[261],{"categories":1160},[],{"categories":1162},[],{"categories":1164},[],{"categories":1166},[313],{"categories":1168},[306],{"categories":1170},[282],{"categories":1172},[303],{"categories":1174},[],{"categories":1176},[261],{"categories":1178},[261],{"categories":1180},[],{"categories":1182},[],{"categories":1184},[264],{"categories":1186},[261],{"categories":1188},[258],{"categories":1190},[],{"categories":1192},[255],{"categories":1194},[261],{"categories":1196},[255],{"categories":1198},[261],{"categories":1200},[313],{"categories":1202},[320],{"categories":1204},[261,303],{"categories":1206},[282],{"categories":1208},[303],{"categories":1210},[],{"categories":1212},[217],{"categories":1214},[303],{"categories":1216},[264],{"categories":1218},[],{"categories":1220},[],{"categories":1222},[],{"categories":1224},[],{"categories":1226},[313],{"categories":1228},[264],{"categories":1230},[264],{"categories":1232},[217],{"categories":1234},[261],{"categories":1236},[261],{"categories":1238},[261],{"categories":1240},[],{"categories":1242},[303],{"categories":1244},[],{"categories":1246},[],{"categories":1248},[264],{"categories":1250},[],{"categories":1252},[],{"categories":1254},[320],{"categories":1256},[320],{"categories":1258},[264],{"categories":1260},[],{"categories":1262},[261],{"categories":1264},[261],{"categories":1266},[313],{"categories":1268},[303],{"categories":1270},[303],{"categories":1272},[264],{"categories":1274},[255],{"categories":1276},[261],{"categories":1278},[303],{"categories":1280},[303],{"categories":1282},[264],{"categories":1284},[264],{"categories":1286},[261],{"categories":1288},[],{"categories":1290},[],{"categories":1292},[261],{"categories":1294},[264],{"categories":1296},[282],{"categories":1298},[313],{"categories":1300},[255],{"categories":1302},[261],{"categories":1304},[],{"categories":1306},[264],{"categories":1308},[264],{"categories":1310},[],{"categories":1312},[255],{"categories":1314},[261],{"categories":1316},[255],{"categories":1318},[255],{"categories":1320},[],{"categories":1322},[],{"categories":1324},[264],{"categories":1326},[264],{"categories":1328},[261],{"categories":1330},[261],{"categories":1332},[282],{"categories":1334},[306],{"categories":1336},[267],{"categories":1338},[282],{"categories":1340},[303],{"categories":1342},[],{"categories":1344},[282],{"categories":1346},[],{"categories":1348},[],{"categories":1350},[],{"categories":1352},[],{"categories":1354},[313],{"categories":1356},[306],{"categories":1358},[],{"categories":1360},[261],{"categories":1362},[261],{"categories":1364},[306],{"categories":1366},[313],{"categories":1368},[],{"categories":1370},[],{"categories":1372},[264],{"categories":1374},[282],{"categories":1376},[282],{"categories":1378},[264],{"categories":1380},[255],{"categories":1382},[261,217],{"categories":1384},[],{"categories":1386},[303],{"categories":1388},[255],{"categories":1390},[264],{"categories":1392},[303],{"categories":1394},[],{"categories":1396},[264],{"categories":1398},[264],{"categories":1400},[261],{"categories":1402},[320],{"categories":1404},[313],{"categories":1406},[303],{"categories":1408},[],{"categories":1410},[264],{"categories":1412},[261],{"categories":1414},[264],{"categories":1416},[264],{"categories":1418},[264],{"categories":1420},[320],{"categories":1422},[264],{"categories":1424},[261],{"categories":1426},[],{"categories":1428},[320],{"categories":1430},[282],{"categories":1432},[264],{"categories":1434},[],{"categories":1436},[],{"categories":1438},[261],{"categories":1440},[264],{"categories":1442},[282],{"categories":1444},[264],{"categories":1446},[],{"categories":1448},[],{"categories":1450},[],{"categories":1452},[264],{"categories":1454},[],{"categories":1456},[],{"categories":1458},[306],{"categories":1460},[261],{"categories":1462},[306],{"categories":1464},[282],{"categories":1466},[261],{"categories":1468},[261],{"categories":1470},[264],{"categories":1472},[261],{"categories":1474},[],{"categories":1476},[],{"categories":1478},[217],{"categories":1480},[],{"categories":1482},[],{"categories":1484},[255],{"categories":1486},[],{"categories":1488},[],{"categories":1490},[],{"categories":1492},[],{"categories":1494},[313],{"categories":1496},[282],{"categories":1498},[320],{"categories":1500},[258],{"categories":1502},[261],{"categories":1504},[261],{"categories":1506},[258],{"categories":1508},[],{"categories":1510},[303],{"categories":1512},[264],{"categories":1514},[258],{"categories":1516},[261],{"categories":1518},[261],{"categories":1520},[255],{"categories":1522},[],{"categories":1524},[255],{"categories":1526},[261],{"categories":1528},[320],{"categories":1530},[264],{"categories":1532},[282],{"categories":1534},[258],{"categories":1536},[261],{"categories":1538},[264],{"categories":1540},[],{"categories":1542},[261],{"categories":1544},[255],{"categories":1546},[261],{"categories":1548},[],{"categories":1550},[282],{"categories":1552},[261],{"categories":1554},[],{"categories":1556},[258],{"categories":1558},[261],{"categories":1560},[],{"categories":1562},[],{"categories":1564},[],{"categories":1566},[261],{"categories":1568},[],{"categories":1570},[217],{"categories":1572},[261],{"categories":1574},[],{"categories":1576},[261],{"categories":1578},[261],{"categories":1580},[261],{"categories":1582},[261,217],{"categories":1584},[261],{"categories":1586},[261],{"categories":1588},[303],{"categories":1590},[264],{"categories":1592},[],{"categories":1594},[264],{"categories":1596},[261],{"categories":1598},[261],{"categories":1600},[261],{"categories":1602},[255],{"categories":1604},[255],{"categories":1606},[313],{"categories":1608},[303],{"categories":1610},[264],{"categories":1612},[],{"categories":1614},[261],{"categories":1616},[282],{"categories":1618},[261],{"categories":1620},[258],{"categories":1622},[],{"categories":1624},[217],{"categories":1626},[303],{"categories":1628},[303],{"categories":1630},[264],{"categories":1632},[282],{"categories":1634},[264],{"categories":1636},[261],{"categories":1638},[],{"categories":1640},[261],{"categories":1642},[],{"categories":1644},[],{"categories":1646},[261],{"categories":1648},[261],{"categories":1650},[261],{"categories":1652},[264],{"categories":1654},[261],{"categories":1656},[],{"categories":1658},[306],{"categories":1660},[264],{"categories":1662},[],{"categories":1664},[],{"categories":1666},[261],{"categories":1668},[282],{"categories":1670},[],{"categories":1672},[303],{"categories":1674},[217],{"categories":1676},[282],{"categories":1678},[313],{"categories":1680},[313],{"categories":1682},[282],{"categories":1684},[282],{"categories":1686},[217],{"categories":1688},[],{"categories":1690},[282],{"categories":1692},[261],{"categories":1694},[255],{"categories":1696},[282],{"categories":1698},[],{"categories":1700},[306],{"categories":1702},[282],{"categories":1704},[313],{"categories":1706},[282],{"categories":1708},[217],{"categories":1710},[261],{"categories":1712},[261],{"categories":1714},[],{"categories":1716},[258],{"categories":1718},[],{"categories":1720},[],{"categories":1722},[261],{"categories":1724},[261],{"categories":1726},[261],{"categories":1728},[261],{"categories":1730},[],{"categories":1732},[306],{"categories":1734},[255],{"categories":1736},[],{"categories":1738},[261],{"categories":1740},[261],{"categories":1742},[217],{"categories":1744},[217],{"categories":1746},[],{"categories":1748},[264],{"categories":1750},[282],{"categories":1752},[282],{"categories":1754},[261],{"categories":1756},[264],{"categories":1758},[],{"categories":1760},[303],{"categories":1762},[261],{"categories":1764},[261],{"categories":1766},[],{"categories":1768},[],{"categories":1770},[217],{"categories":1772},[261],{"categories":1774},[313],{"categories":1776},[258],{"categories":1778},[261],{"categories":1780},[],{"categories":1782},[264],{"categories":1784},[255],{"categories":1786},[255],{"categories":1788},[],{"categories":1790},[261],{"categories":1792},[303],{"categories":1794},[264],{"categories":1796},[],{"categories":1798},[261],{"categories":1800},[261],{"categories":1802},[264],{"categories":1804},[],{"categories":1806},[264],{"categories":1808},[313],{"categories":1810},[],{"categories":1812},[261],{"categories":1814},[],{"categories":1816},[261],{"categories":1818},[],{"categories":1820},[261],{"categories":1822},[261],{"categories":1824},[],{"categories":1826},[261],{"categories":1828},[282],{"categories":1830},[261],{"categories":1832},[261],{"categories":1834},[255],{"categories":1836},[261],{"categories":1838},[282],{"categories":1840},[264],{"categories":1842},[],{"categories":1844},[261],{"categories":1846},[320],{"categories":1848},[],{"categories":1850},[],{"categories":1852},[],{"categories":1854},[255],{"categories":1856},[282],{"categories":1858},[264],{"categories":1860},[261],{"categories":1862},[303],{"categories":1864},[264],{"categories":1866},[],{"categories":1868},[264],{"categories":1870},[],{"categories":1872},[261],{"categories":1874},[264],{"categories":1876},[261],{"categories":1878},[],{"categories":1880},[261],{"categories":1882},[261],{"categories":1884},[282],{"categories":1886},[303],{"categories":1888},[264],{"categories":1890},[303],{"categories":1892},[258],{"categories":1894},[],{"categories":1896},[],{"categories":1898},[261],{"categories":1900},[255],{"categories":1902},[282],{"categories":1904},[],{"categories":1906},[],{"categories":1908},[313],{"categories":1910},[303],{"categories":1912},[],{"categories":1914},[261],{"categories":1916},[],{"categories":1918},[320],{"categories":1920},[261],{"categories":1922},[217],{"categories":1924},[313],{"categories":1926},[],{"categories":1928},[264],{"categories":1930},[261],{"categories":1932},[264],{"categories":1934},[264],{"categories":1936},[261],{"categories":1938},[],{"categories":1940},[255],{"categories":1942},[261],{"categories":1944},[258],{"categories":1946},[313],{"categories":1948},[303],{"categories":1950},[],{"categories":1952},[],{"categories":1954},[],{"categories":1956},[264],{"categories":1958},[303],{"categories":1960},[282],{"categories":1962},[261],{"categories":1964},[282],{"categories":1966},[303],{"categories":1968},[],{"categories":1970},[303],{"categories":1972},[282],{"categories":1974},[258],{"categories":1976},[261],{"categories":1978},[282],{"categories":1980},[320],{"categories":1982},[],{"categories":1984},[],{"categories":1986},[306],{"categories":1988},[261,313],{"categories":1990},[282],{"categories":1992},[261],{"categories":1994},[264],{"categories":1996},[264],{"categories":1998},[261],{"categories":2000},[],{"categories":2002},[313],{"categories":2004},[261],{"categories":2006},[306],{"categories":2008},[264],{"categories":2010},[320],{"categories":2012},[217],{"categories":2014},[],{"categories":2016},[255],{"categories":2018},[264],{"categories":2020},[264],{"categories":2022},[313],{"categories":2024},[261],{"categories":2026},[261],{"categories":2028},[],{"categories":2030},[],{"categories":2032},[],{"categories":2034},[217],{"categories":2036},[282],{"categories":2038},[261],{"categories":2040},[261],{"categories":2042},[261],{"categories":2044},[],{"categories":2046},[306],{"categories":2048},[258],{"categories":2050},[],{"categories":2052},[264],{"categories":2054},[217],{"categories":2056},[],{"categories":2058},[303],{"categories":2060},[303],{"categories":2062},[],{"categories":2064},[313],{"categories":2066},[303],{"categories":2068},[261],{"categories":2070},[],{"categories":2072},[282],{"categories":2074},[261],{"categories":2076},[303],{"categories":2078},[264],{"categories":2080},[282],{"categories":2082},[],{"categories":2084},[264],{"categories":2086},[303],{"categories":2088},[261],{"categories":2090},[],{"categories":2092},[261],{"categories":2094},[261],{"categories":2096},[217],{"categories":2098},[282],{"categories":2100},[306],{"categories":2102},[306],{"categories":2104},[],{"categories":2106},[],{"categories":2108},[],{"categories":2110},[264],{"categories":2112},[313],{"categories":2114},[313],{"categories":2116},[],{"categories":2118},[],{"categories":2120},[261],{"categories":2122},[],{"categories":2124},[264],{"categories":2126},[261],{"categories":2128},[],{"categories":2130},[261],{"categories":2132},[258],{"categories":2134},[261],{"categories":2136},[320],{"categories":2138},[264],{"categories":2140},[261],{"categories":2142},[313],{"categories":2144},[282],{"categories":2146},[264],{"categories":2148},[],{"categories":2150},[282],{"categories":2152},[264],{"categories":2154},[264],{"categories":2156},[],{"categories":2158},[258],{"categories":2160},[264],{"categories":2162},[],{"categories":2164},[261],{"categories":2166},[255],{"categories":2168},[282],{"categories":2170},[217],{"categories":2172},[264],{"categories":2174},[264],{"categories":2176},[255],{"categories":2178},[261],{"categories":2180},[],{"categories":2182},[],{"categories":2184},[303],{"categories":2186},[261,258],{"categories":2188},[],{"categories":2190},[255],{"categories":2192},[306],{"categories":2194},[261],{"categories":2196},[313],{"categories":2198},[261],{"categories":2200},[264],{"categories":2202},[261],{"categories":2204},[261],{"categories":2206},[282],{"categories":2208},[264],{"categories":2210},[],{"categories":2212},[],{"categories":2214},[264],{"categories":2216},[261],{"categories":2218},[217],{"categories":2220},[],{"categories":2222},[261],{"categories":2224},[264],{"categories":2226},[],{"categories":2228},[261],{"categories":2230},[320],{"categories":2232},[306],{"categories":2234},[264],{"categories":2236},[261],{"categories":2238},[217],{"categories":2240},[],{"categories":2242},[261],{"categories":2244},[320],{"categories":2246},[303],{"categories":2248},[261],{"categories":2250},[],{"categories":2252},[320],{"categories":2254},[282],{"categories":2256},[261],{"categories":2258},[261],{"categories":2260},[255],{"categories":2262},[],{"categories":2264},[],{"categories":2266},[303],{"categories":2268},[261],{"categories":2270},[306],{"categories":2272},[320],{"categories":2274},[320],{"categories":2276},[282],{"categories":2278},[],{"categories":2280},[],{"categories":2282},[261],{"categories":2284},[],{"categories":2286},[261,313],{"categories":2288},[282],{"categories":2290},[264],{"categories":2292},[313],{"categories":2294},[261],{"categories":2296},[255],{"categories":2298},[],{"categories":2300},[],{"categories":2302},[255],{"categories":2304},[320],{"categories":2306},[261],{"categories":2308},[],{"categories":2310},[303,261],{"categories":2312},[217],{"categories":2314},[255],{"categories":2316},[],{"categories":2318},[258],{"categories":2320},[258],{"categories":2322},[261],{"categories":2324},[313],{"categories":2326},[264],{"categories":2328},[282],{"categories":2330},[320],{"categories":2332},[303],{"categories":2334},[261],{"categories":2336},[261],{"categories":2338},[261],{"categories":2340},[255],{"categories":2342},[261],{"categories":2344},[264],{"categories":2346},[282],{"categories":2348},[],{"categories":2350},[],{"categories":2352},[306],{"categories":2354},[313],{"categories":2356},[261],{"categories":2358},[303],{"categories":2360},[306],{"categories":2362},[261],{"categories":2364},[261],{"categories":2366},[264],{"categories":2368},[264],{"categories":2370},[261,258],{"categories":2372},[],{"categories":2374},[303],{"categories":2376},[],{"categories":2378},[261],{"categories":2380},[282],{"categories":2382},[255],{"categories":2384},[255],{"categories":2386},[264],{"categories":2388},[261],{"categories":2390},[258],{"categories":2392},[313],{"categories":2394},[320],{"categories":2396},[],{"categories":2398},[282],{"categories":2400},[261],{"categories":2402},[261],{"categories":2404},[282],{"categories":2406},[313],{"categories":2408},[261],{"categories":2410},[264],{"categories":2412},[282],{"categories":2414},[261],{"categories":2416},[303],{"categories":2418},[261],{"categories":2420},[261],{"categories":2422},[217],{"categories":2424},[267],{"categories":2426},[264],{"categories":2428},[261],{"categories":2430},[282],{"categories":2432},[264],{"categories":2434},[320],{"categories":2436},[261],{"categories":2438},[],{"categories":2440},[261],{"categories":2442},[],{"categories":2444},[],{"categories":2446},[],{"categories":2448},[258],{"categories":2450},[261],{"categories":2452},[264],{"categories":2454},[282],{"categories":2456},[282],{"categories":2458},[282],{"categories":2460},[282],{"categories":2462},[],{"categories":2464},[255],{"categories":2466},[264],{"categories":2468},[282],{"categories":2470},[255],{"categories":2472},[264],{"categories":2474},[261],{"categories":2476},[261,264],{"categories":2478},[264],{"categories":2480},[217],{"categories":2482},[282],{"categories":2484},[282],{"categories":2486},[264],{"categories":2488},[261],{"categories":2490},[],{"categories":2492},[282],{"categories":2494},[320],{"categories":2496},[255],{"categories":2498},[261],{"categories":2500},[261],{"categories":2502},[],{"categories":2504},[313],{"categories":2506},[],{"categories":2508},[255],{"categories":2510},[264],{"categories":2512},[282],{"categories":2514},[261],{"categories":2516},[282],{"categories":2518},[255],{"categories":2520},[282],{"categories":2522},[282],{"categories":2524},[],{"categories":2526},[258],{"categories":2528},[264],{"categories":2530},[282],{"categories":2532},[282],{"categories":2534},[282],{"categories":2536},[282],{"categories":2538},[282],{"categories":2540},[282],{"categories":2542},[282],{"categories":2544},[282],{"categories":2546},[282],{"categories":2548},[282],{"categories":2550},[306],{"categories":2552},[255],{"categories":2554},[261],{"categories":2556},[261],{"categories":2558},[],{"categories":2560},[261,255],{"categories":2562},[],{"categories":2564},[264],{"categories":2566},[282],{"categories":2568},[264],{"categories":2570},[261],{"categories":2572},[261],{"categories":2574},[261],{"categories":2576},[261],{"categories":2578},[261],{"categories":2580},[264],{"categories":2582},[258],{"categories":2584},[303],{"categories":2586},[282],{"categories":2588},[261],{"categories":2590},[],{"categories":2592},[],{"categories":2594},[264],{"categories":2596},[303],{"categories":2598},[261],{"categories":2600},[],{"categories":2602},[],{"categories":2604},[320],{"categories":2606},[261],{"categories":2608},[],{"categories":2610},[],{"categories":2612},[255],{"categories":2614},[258],{"categories":2616},[261],{"categories":2618},[258],{"categories":2620},[303],{"categories":2622},[],{"categories":2624},[282],{"categories":2626},[],{"categories":2628},[303],{"categories":2630},[261],{"categories":2632},[320],{"categories":2634},[],{"categories":2636},[320],{"categories":2638},[],{"categories":2640},[],{"categories":2642},[264],{"categories":2644},[],{"categories":2646},[258],{"categories":2648},[255],{"categories":2650},[303],{"categories":2652},[313],{"categories":2654},[],{"categories":2656},[],{"categories":2658},[261],{"categories":2660},[255],{"categories":2662},[320],{"categories":2664},[],{"categories":2666},[264],{"categories":2668},[264],{"categories":2670},[282],{"categories":2672},[261],{"categories":2674},[264],{"categories":2676},[261],{"categories":2678},[264],{"categories":2680},[261],{"categories":2682},[267],{"categories":2684},[282],{"categories":2686},[],{"categories":2688},[320],{"categories":2690},[313],{"categories":2692},[264],{"categories":2694},[],{"categories":2696},[261],{"categories":2698},[264],{"categories":2700},[258],{"categories":2702},[255],{"categories":2704},[261],{"categories":2706},[303],{"categories":2708},[313],{"categories":2710},[313],{"categories":2712},[261],{"categories":2714},[306],{"categories":2716},[261],{"categories":2718},[264],{"categories":2720},[258],{"categories":2722},[264],{"categories":2724},[261],{"categories":2726},[261],{"categories":2728},[264],{"categories":2730},[282],{"categories":2732},[],{"categories":2734},[255],{"categories":2736},[261],{"categories":2738},[264],{"categories":2740},[261],{"categories":2742},[261],{"categories":2744},[],{"categories":2746},[303],{"categories":2748},[258],{"categories":2750},[282],{"categories":2752},[261],{"categories":2754},[261],{"categories":2756},[303],{"categories":2758},[320],{"categories":2760},[306],{"categories":2762},[261],{"categories":2764},[282],{"categories":2766},[261],{"categories":2768},[264],{"categories":2770},[217],{"categories":2772},[261],{"categories":2774},[264],{"categories":2776},[306],{"categories":2778},[],{"categories":2780},[264],{"categories":2782},[313],{"categories":2784},[303],{"categories":2786},[261],{"categories":2788},[255],{"categories":2790},[258],{"categories":2792},[313],{"categories":2794},[],{"categories":2796},[264],{"categories":2798},[261],{"categories":2800},[],{"categories":2802},[282],{"categories":2804},[],{"categories":2806},[282],{"categories":2808},[261],{"categories":2810},[264],{"categories":2812},[264],{"categories":2814},[264],{"categories":2816},[],{"categories":2818},[],{"categories":2820},[261],{"categories":2822},[261],{"categories":2824},[],{"categories":2826},[303],{"categories":2828},[264],{"categories":2830},[320],{"categories":2832},[255],{"categories":2834},[],{"categories":2836},[],{"categories":2838},[282],{"categories":2840},[313],{"categories":2842},[261],{"categories":2844},[261],{"categories":2846},[261],{"categories":2848},[313],{"categories":2850},[282],{"categories":2852},[303],{"categories":2854},[261],{"categories":2856},[261],{"categories":2858},[261],{"categories":2860},[282],{"categories":2862},[261],{"categories":2864},[282],{"categories":2866},[264],{"categories":2868},[264],{"categories":2870},[313],{"categories":2872},[264],{"categories":2874},[261],{"categories":2876},[313],{"categories":2878},[303],{"categories":2880},[],{"categories":2882},[264],{"categories":2884},[],{"categories":2886},[],{"categories":2888},[],{"categories":2890},[258],{"categories":2892},[261],{"categories":2894},[264],{"categories":2896},[255],{"categories":2898},[264],{"categories":2900},[320],{"categories":2902},[],{"categories":2904},[264],{"categories":2906},[],{"categories":2908},[255],{"categories":2910},[264],{"categories":2912},[],{"categories":2914},[264],{"categories":2916},[261],{"categories":2918},[282],{"categories":2920},[261],{"categories":2922},[264],{"categories":2924},[282],{"categories":2926},[264],{"categories":2928},[313],{"categories":2930},[303],{"categories":2932},[255],{"categories":2934},[],{"categories":2936},[264],{"categories":2938},[303],{"categories":2940},[217],{"categories":2942},[282],{"categories":2944},[261],{"categories":2946},[303],{"categories":2948},[255],{"categories":2950},[],{"categories":2952},[264],{"categories":2954},[264],{"categories":2956},[261],{"categories":2958},[],{"categories":2960},[264],{"categories":2962},[267],{"categories":2964},[282],{"categories":2966},[264],{"categories":2968},[258],{"categories":2970},[],{"categories":2972},[261],{"categories":2974},[267],{"categories":2976},[261],{"categories":2978},[264],{"categories":2980},[282],{"categories":2982},[255],{"categories":2984},[217],{"categories":2986},[261],{"categories":2988},[261],{"categories":2990},[261],{"categories":2992},[282],{"categories":2994},[258],{"categories":2996},[261],{"categories":2998},[303],{"categories":3000},[282],{"categories":3002},[217],{"categories":3004},[261],{"categories":3006},[],{"categories":3008},[],{"categories":3010},[217],{"categories":3012},[306],{"categories":3014},[264],{"categories":3016},[264],{"categories":3018},[282],{"categories":3020},[261],{"categories":3022},[255],{"categories":3024},[303],{"categories":3026},[264],{"categories":3028},[261],{"categories":3030},[320],{"categories":3032},[261],{"categories":3034},[264],{"categories":3036},[],{"categories":3038},[261],{"categories":3040},[261],{"categories":3042},[282],{"categories":3044},[255],{"categories":3046},[],{"categories":3048},[261],{"categories":3050},[261],{"categories":3052},[313],{"categories":3054},[303],{"categories":3056},[261,264],{"categories":3058},[320,258],{"categories":3060},[261],{"categories":3062},[],{"categories":3064},[264],{"categories":3066},[],{"categories":3068},[313],{"categories":3070},[261],{"categories":3072},[282],{"categories":3074},[],{"categories":3076},[264],{"categories":3078},[],{"categories":3080},[303],{"categories":3082},[264],{"categories":3084},[255],{"categories":3086},[264],{"categories":3088},[261],{"categories":3090},[217],{"categories":3092},[320],{"categories":3094},[258],{"categories":3096},[258],{"categories":3098},[255],{"categories":3100},[255],{"categories":3102},[261],{"categories":3104},[264],{"categories":3106},[261],{"categories":3108},[261],{"categories":3110},[255],{"categories":3112},[261],{"categories":3114},[320],{"categories":3116},[282],{"categories":3118},[261],{"categories":3120},[264],{"categories":3122},[261],{"categories":3124},[],{"categories":3126},[313],{"categories":3128},[],{"categories":3130},[264],{"categories":3132},[255],{"categories":3134},[],{"categories":3136},[217],{"categories":3138},[261],{"categories":3140},[],{"categories":3142},[282],{"categories":3144},[264],{"categories":3146},[313],{"categories":3148},[261],{"categories":3150},[264],{"categories":3152},[313],{"categories":3154},[264],{"categories":3156},[282],{"categories":3158},[255],{"categories":3160},[282],{"categories":3162},[313],{"categories":3164},[261],{"categories":3166},[303],{"categories":3168},[261],{"categories":3170},[261],{"categories":3172},[261],{"categories":3174},[261],{"categories":3176},[264],{"categories":3178},[261],{"categories":3180},[264],{"categories":3182},[261],{"categories":3184},[255],{"categories":3186},[261],{"categories":3188},[264],{"categories":3190},[303],{"categories":3192},[255],{"categories":3194},[264],{"categories":3196},[303],{"categories":3198},[],{"categories":3200},[261],{"categories":3202},[261],{"categories":3204},[313],{"categories":3206},[],{"categories":3208},[264],{"categories":3210},[320],{"categories":3212},[261],{"categories":3214},[282],{"categories":3216},[320],{"categories":3218},[264],{"categories":3220},[258],{"categories":3222},[258],{"categories":3224},[261],{"categories":3226},[255],{"categories":3228},[],{"categories":3230},[261],{"categories":3232},[],{"categories":3234},[255],{"categories":3236},[261],{"categories":3238},[264],{"categories":3240},[264],{"categories":3242},[],{"categories":3244},[313],{"categories":3246},[313],{"categories":3248},[320],{"categories":3250},[303],{"categories":3252},[],{"categories":3254},[261],{"categories":3256},[255],{"categories":3258},[261],{"categories":3260},[313],{"categories":3262},[255],{"categories":3264},[282],{"categories":3266},[282],{"categories":3268},[],{"categories":3270},[282],{"categories":3272},[264],{"categories":3274},[303],{"categories":3276},[306],{"categories":3278},[261],{"categories":3280},[],{"categories":3282},[282],{"categories":3284},[313],{"categories":3286},[258],{"categories":3288},[261],{"categories":3290},[255],{"categories":3292},[217],{"categories":3294},[255],{"categories":3296},[],{"categories":3298},[],{"categories":3300},[282],{"categories":3302},[],{"categories":3304},[264],{"categories":3306},[264],{"categories":3308},[264],{"categories":3310},[],{"categories":3312},[261],{"categories":3314},[],{"categories":3316},[282],{"categories":3318},[255],{"categories":3320},[303],{"categories":3322},[261],{"categories":3324},[282],{"categories":3326},[282],{"categories":3328},[],{"categories":3330},[282],{"categories":3332},[255],{"categories":3334},[261],{"categories":3336},[],{"categories":3338},[264],{"categories":3340},[264],{"categories":3342},[255],{"categories":3344},[],{"categories":3346},[],{"categories":3348},[],{"categories":3350},[303],{"categories":3352},[264],{"categories":3354},[261],{"categories":3356},[],{"categories":3358},[],{"categories":3360},[],{"categories":3362},[303],{"categories":3364},[],{"categories":3366},[255],{"categories":3368},[],{"categories":3370},[],{"categories":3372},[303],{"categories":3374},[261],{"categories":3376},[282],{"categories":3378},[],{"categories":3380},[320],{"categories":3382},[282],{"categories":3384},[320],{"categories":3386},[261],{"categories":3388},[],{"categories":3390},[],{"categories":3392},[264],{"categories":3394},[],{"categories":3396},[],{"categories":3398},[264],{"categories":3400},[261],{"categories":3402},[],{"categories":3404},[264],{"categories":3406},[282],{"categories":3408},[320],{"categories":3410},[306],{"categories":3412},[264],{"categories":3414},[264],{"categories":3416},[],{"categories":3418},[],{"categories":3420},[],{"categories":3422},[282],{"categories":3424},[],{"categories":3426},[],{"categories":3428},[303],{"categories":3430},[255],{"categories":3432},[],{"categories":3434},[258],{"categories":3436},[320],{"categories":3438},[261],{"categories":3440},[313],{"categories":3442},[255],{"categories":3444},[306],{"categories":3446},[258],{"categories":3448},[313],{"categories":3450},[],{"categories":3452},[],{"categories":3454},[264],{"categories":3456},[255],{"categories":3458},[303],{"categories":3460},[255],{"categories":3462},[264],{"categories":3464},[217],{"categories":3466},[264],{"categories":3468},[],{"categories":3470},[261],{"categories":3472},[282],{"categories":3474},[313],{"categories":3476},[],{"categories":3478},[303],{"categories":3480},[282],{"categories":3482},[255],{"categories":3484},[264],{"categories":3486},[261],{"categories":3488},[258],{"categories":3490},[264,217],{"categories":3492},[264],{"categories":3494},[313],{"categories":3496},[261],{"categories":3498},[306],{"categories":3500},[320],{"categories":3502},[264],{"categories":3504},[],{"categories":3506},[264],{"categories":3508},[261],{"categories":3510},[258],{"categories":3512},[],{"categories":3514},[],{"categories":3516},[261],{"categories":3518},[306],{"categories":3520},[261],{"categories":3522},[],{"categories":3524},[282],{"categories":3526},[],{"categories":3528},[282],{"categories":3530},[313],{"categories":3532},[264],{"categories":3534},[261],{"categories":3536},[320],{"categories":3538},[313],{"categories":3540},[],{"categories":3542},[282],{"categories":3544},[261],{"categories":3546},[],{"categories":3548},[261],{"categories":3550},[264],{"categories":3552},[261],{"categories":3554},[264],{"categories":3556},[261],{"categories":3558},[261],{"categories":3560},[261],{"categories":3562},[261],{"categories":3564},[258],{"categories":3566},[],{"categories":3568},[267],{"categories":3570},[282],{"categories":3572},[261],{"categories":3574},[],{"categories":3576},[313],{"categories":3578},[261],{"categories":3580},[261],{"categories":3582},[264],{"categories":3584},[282],{"categories":3586},[261],{"categories":3588},[261],{"categories":3590},[258],{"categories":3592},[264],{"categories":3594},[303],{"categories":3596},[],{"categories":3598},[306],{"categories":3600},[261],{"categories":3602},[],{"categories":3604},[282],{"categories":3606},[320],{"categories":3608},[],{"categories":3610},[],{"categories":3612},[282],{"categories":3614},[282],{"categories":3616},[320],{"categories":3618},[255],{"categories":3620},[264],{"categories":3622},[264],{"categories":3624},[261],{"categories":3626},[258],{"categories":3628},[],{"categories":3630},[],{"categories":3632},[282],{"categories":3634},[306],{"categories":3636},[313],{"categories":3638},[264],{"categories":3640},[303],{"categories":3642},[306],{"categories":3644},[306],{"categories":3646},[],{"categories":3648},[282],{"categories":3650},[261],{"categories":3652},[261],{"categories":3654},[313],{"categories":3656},[],{"categories":3658},[282],{"categories":3660},[282],{"categories":3662},[282],{"categories":3664},[],{"categories":3666},[264],{"categories":3668},[261],{"categories":3670},[],{"categories":3672},[255],{"categories":3674},[258],{"categories":3676},[],{"categories":3678},[261],{"categories":3680},[261],{"categories":3682},[],{"categories":3684},[313],{"categories":3686},[],{"categories":3688},[],{"categories":3690},[],{"categories":3692},[],{"categories":3694},[261],{"categories":3696},[282],{"categories":3698},[],{"categories":3700},[],{"categories":3702},[261],{"categories":3704},[261],{"categories":3706},[261],{"categories":3708},[306],{"categories":3710},[261],{"categories":3712},[306],{"categories":3714},[],{"categories":3716},[306],{"categories":3718},[306],{"categories":3720},[217],{"categories":3722},[264],{"categories":3724},[313],{"categories":3726},[],{"categories":3728},[],{"categories":3730},[306],{"categories":3732},[313],{"categories":3734},[313],{"categories":3736},[313],{"categories":3738},[],{"categories":3740},[255],{"categories":3742},[313],{"categories":3744},[313],{"categories":3746},[255],{"categories":3748},[313],{"categories":3750},[258],{"categories":3752},[313],{"categories":3754},[313],{"categories":3756},[313],{"categories":3758},[306],{"categories":3760},[282],{"categories":3762},[282],{"categories":3764},[261],{"categories":3766},[313],{"categories":3768},[306],{"categories":3770},[217],{"categories":3772},[306],{"categories":3774},[306],{"categories":3776},[306],{"categories":3778},[],{"categories":3780},[258],{"categories":3782},[],{"categories":3784},[217],{"categories":3786},[313],{"categories":3788},[313],{"categories":3790},[313],{"categories":3792},[264],{"categories":3794},[282,258],{"categories":3796},[306],{"categories":3798},[],{"categories":3800},[],{"categories":3802},[306],{"categories":3804},[],{"categories":3806},[306],{"categories":3808},[282],{"categories":3810},[264],{"categories":3812},[],{"categories":3814},[313],{"categories":3816},[261],{"categories":3818},[303],{"categories":3820},[],{"categories":3822},[261],{"categories":3824},[],{"categories":3826},[282],{"categories":3828},[255],{"categories":3830},[306],{"categories":3832},[],{"categories":3834},[313],{"categories":3836},[282],[3838,3907,3976,4037],{"id":3839,"title":3840,"ai":3841,"body":3846,"categories":3882,"created_at":218,"date_modified":218,"description":48,"extension":219,"faq":218,"featured":220,"kicker_label":218,"meta":3883,"navigation":103,"path":3895,"published_at":3896,"question":218,"scraped_at":3896,"seo":3897,"sitemap":3898,"source_id":3899,"source_name":3900,"source_type":242,"source_url":3901,"stem":3902,"tags":3903,"thumbnail_url":218,"tldr":3904,"tweet":218,"unknown_tags":3905,"__hash__":3906},"summaries\u002Fsummaries\u002Fa2a811b50a4c64f5-mrc-resilient-networking-for-100k-gpu-ai-training-summary.md","MRC: Resilient Networking for 100K+ GPU AI Training",{"provider":7,"model":8,"input_tokens":3842,"output_tokens":3843,"processing_time_ms":3844,"cost_usd":3845},9014,2044,25377,0.0028023,{"type":14,"value":3847,"toc":3876},[3848,3852,3855,3859,3862,3866,3869,3873],[17,3849,3851],{"id":3850},"multi-plane-topologies-slash-switch-tiers-and-power-for-massive-clusters","Multi-Plane Topologies Slash Switch Tiers and Power for Massive Clusters",[22,3853,3854],{},"Traditional 800Gb\u002Fs networks require three or four tiers of switches to connect over 100,000 GPUs, increasing power use, failure points, and cost. MRC splits each 800Gb\u002Fs interface into eight 100Gb\u002Fs links, creating eight parallel 'planes' that connect to separate switches. A 64-port 800Gb\u002Fs switch now handles 512 ports at 100Gb\u002Fs, enabling full connectivity for 131,000 GPUs using only two tiers. This design boosts path diversity—keeping more traffic local to Tier 0 switches—while cutting components, power, and cost compared to single-plane setups. Without changes, single-path flows (like classic RoCE) still congest links as flows collide, especially in AI's collective communications where worst-case latency stalls synchronous training.",[17,3856,3858],{"id":3857},"packet-spraying-and-srv6-eliminate-congestion-and-dynamic-routing","Packet Spraying and SRv6 Eliminate Congestion and Dynamic Routing",[22,3860,3861],{},"MRC sprays packets from a single transfer across hundreds of paths spanning all planes, using final memory addresses for out-of-order reassembly at the destination. Adaptive load-balancing monitors paths: congestion triggers path swaps, packet loss retires the path (with probes for recovery), and 'packet trimming' at switches forwards headers only during destination congestion to prompt retransmits without false failure alarms. This achieves microsecond failure detection and rerouting, versus seconds for traditional fabrics. MRC replaces BGP dynamic routing with static SRv6 source routing: senders embed full switch ID sequences in IPv6 addresses. Switches shift addresses and follow pre-configured static tables, blindly forwarding without recomputing routes. Failures simply retire paths at endpoints, simplifying control planes and eliminating routing bugs from switch software.",[17,3863,3865],{"id":3864},"production-impact-zero-measurable-downtime-amid-constant-failures","Production Impact: Zero-Measurable Downtime Amid Constant Failures",[22,3867,3868],{},"In OpenAI's NVIDIA GB200 supercomputers (including OCI's Abilene Stargate site and Microsoft's Fairwater), MRC handles millions of links with frequent flaps—multiple per minute between tiers—yet synchronous pretraining jobs show no measurable impact, allowing deferred repairs. Rebooting four Tier-1 switches or repairing links during jobs requires no coordination; MRC avoids bad paths automatically. Real training data shows quick recovery from full T1 switch loss with temporary slowdowns far less than physical capacity loss (e.g., one failed port on an 8-port interface reduces max rate by 1\u002F8th but sustains better effective throughput via path recalculation). Multi-job clusters avoid inter-job interference due to core-wide congestion elimination, maximizing GPU utilization for frontier models like those powering ChatGPT (900M weekly users).",[17,3870,3872],{"id":3871},"strategic-wins-simpler-stacks-for-stargate-scale-compute","Strategic Wins: Simpler Stacks for Stargate-Scale Compute",[22,3874,3875],{},"MRC delivers three edges: two-tier multi-plane redundancy with lower power; zero core congestion for consistent flow throughput in sync training; and SRv6 for instant failure bypass via static planes. Deployed with AMD, Broadcom, Intel, Microsoft, NVIDIA hardware, it's released via Open Compute Project for industry adoption, supporting OpenAI's compute strategy of shared standards to scale AI infrastructure efficiently.",{"title":48,"searchDepth":82,"depth":82,"links":3877},[3878,3879,3880,3881],{"id":3850,"depth":82,"text":3851},{"id":3857,"depth":82,"text":3858},{"id":3864,"depth":82,"text":3865},{"id":3871,"depth":82,"text":3872},[217],{"content_references":3884,"triage":3892},[3885,3888],{"type":224,"title":3886,"url":3887,"context":227},"OCP MRC 1.0","https:\u002F\u002Fwww.opencompute.org\u002Fdocuments\u002Focp-mrc-1-0-pdf",{"type":3889,"title":3890,"url":3891,"context":227},"paper","Resilient AI Supercomputer Networking using MRC and SRv6","https:\u002F\u002Fcdn.openai.com\u002Fpdf\u002Fresilient-ai-supercomputer-networking-using-mrc-and-srv6.pdf",{"relevance":88,"novelty":88,"quality":94,"actionability":82,"composite":3893,"reasoning":3894},3.05,"Category: DevOps & Cloud. The article discusses the MRC protocol's innovative networking solutions for AI training, which could be relevant for those building AI-powered products. However, it lacks direct actionable insights for the audience, focusing more on technical specifications than practical applications.","\u002Fsummaries\u002Fa2a811b50a4c64f5-mrc-resilient-networking-for-100k-gpu-ai-training-summary","2026-05-11 15:04:27",{"title":3840,"description":48},{"loc":3895},"a2a811b50a4c64f5","OpenAI News","https:\u002F\u002Fopenai.com\u002Findex\u002Fmrc-supercomputer-networking","summaries\u002Fa2a811b50a4c64f5-mrc-resilient-networking-for-100k-gpu-ai-training-summary",[246,247,248],"OpenAI's MRC protocol uses multi-plane topologies and packet spraying across hundreds of paths with SRv6 source routing to eliminate congestion, route around failures in microseconds, and connect 131k GPUs with just two switch tiers, enabling non-stop frontier model training.",[],"BYXvfLzxxajQIir95xuUTVdTfvID4wPt3TOVHNxrCSU",{"id":3908,"title":3909,"ai":3910,"body":3915,"categories":3952,"created_at":218,"date_modified":218,"description":48,"extension":219,"faq":218,"featured":220,"kicker_label":218,"meta":3953,"navigation":103,"path":3963,"published_at":3964,"question":218,"scraped_at":3965,"seo":3966,"sitemap":3967,"source_id":3968,"source_name":3969,"source_type":242,"source_url":3970,"stem":3971,"tags":3972,"thumbnail_url":218,"tldr":3973,"tweet":218,"unknown_tags":3974,"__hash__":3975},"summaries\u002Fsummaries\u002F30072e6e8b386729-mrc-openai-s-protocol-for-resilient-ai-training-ne-summary.md","MRC: OpenAI's Protocol for Resilient AI Training Networks",{"provider":7,"model":8,"input_tokens":3911,"output_tokens":3912,"processing_time_ms":3913,"cost_usd":3914},8465,1915,20569,0.00214365,{"type":14,"value":3916,"toc":3947},[3917,3921,3924,3927,3930,3934,3937,3940,3944],[17,3918,3920],{"id":3919},"multipath-mechanisms-eliminate-congestion-and-enable-fast-recovery","Multipath Mechanisms Eliminate Congestion and Enable Fast Recovery",[22,3922,3923],{},"In large AI training clusters, network congestion, link failures, and jitter cause GPU idle time, amplifying costs as clusters scale to millions of data transfers per step. MRC builds on RoCEv2 for hardware-accelerated RDMA over Ethernet and SRv6 for static source routing, shifting intelligence to NICs while switches follow pre-configured paths blindly. This avoids interference from dynamic routing.",[22,3925,3926],{},"Adaptive packet spraying distributes packets across hundreds of paths at the NIC level, achieving higher bandwidth, reduced tail latency, and packet-level load balancing—unlike single-path RoCEv2. For failures, MRC detects issues in microseconds and reroutes: if an 8-port 800Gb\u002Fs NIC loses one port, it drops to 7\u002F8 capacity but recalculates paths instantly, notifies peers to avoid the failed plane, and restores it within a minute upon recovery. Conventional fabrics take seconds to tens of seconds, often crashing jobs; MRC keeps training alive with minimal performance hit.",[22,3928,3929],{},"AMD's NSCC congestion control integrates via UEC specs, preserving RDMA semantics while adding multipath support.",[17,3931,3933],{"id":3932},"multi-plane-architecture-cuts-tiers-costs-and-latency","Multi-Plane Architecture Cuts Tiers, Costs, and Latency",[22,3935,3936],{},"MRC reimagines NICs as multiple smaller links (e.g., one 800Gb\u002Fs interface split into eight 100Gb\u002Fs to eight switches), enabling a two-tier Clos network for 131,000 GPUs versus three-to-four tiers in 800Gb\u002Fs designs. Longest paths cross three switches instead of five-to-seven, slashing latency.",[22,3938,3939],{},"For full bisection bandwidth, this uses 2\u002F3 the optics and 3\u002F5 the switches of three-tier networks, reducing power, cost, and failure blast radius. A tier-1 switch failure (e.g., rebooting four during training) no longer halts jobs.",[17,3941,3943],{"id":3942},"production-on-named-hardware-across-openai-clusters","Production on Named Hardware Across OpenAI Clusters",[22,3945,3946],{},"Deployed on 400\u002F800Gb\u002Fs RDMA NICs like NVIDIA ConnectX-8, AMD Pollara\u002FVulcano, Broadcom Thor Ultra; SRv6 switches include NVIDIA Spectrum-4\u002F5 (Cumulus\u002FSONiC) and Broadcom Tomahawk 5 (Arista EOS). Powers NVIDIA GB200 supercomputers in OpenAI's Stargate (OCI Abilene, TX) and Microsoft's Fairwater (Atlanta\u002FWisconsin), training ChatGPT and Codex models without job interruptions from failures.",{"title":48,"searchDepth":82,"depth":82,"links":3948},[3949,3950,3951],{"id":3919,"depth":82,"text":3920},{"id":3932,"depth":82,"text":3933},{"id":3942,"depth":82,"text":3943},[217],{"content_references":3954,"triage":3961},[3955,3957],{"type":3889,"title":3890,"url":3891,"context":3956},"cited",{"type":224,"title":3958,"url":3959,"context":3960},"MRC Supercomputer Networking Technical Details","https:\u002F\u002Fopenai.com\u002Findex\u002Fmrc-supercomputer-networking\u002F","recommended",{"relevance":88,"novelty":88,"quality":94,"actionability":82,"composite":3893,"reasoning":3962},"Category: AI & LLMs. The article discusses OpenAI's MRC protocol, which is relevant to AI infrastructure but lacks direct applicability for product builders looking for actionable insights. While it presents some new technical details about network optimization for AI training, it does not provide practical steps or frameworks that the audience can implement.","\u002Fsummaries\u002F30072e6e8b386729-mrc-openai-s-protocol-for-resilient-ai-training-ne-summary","2026-05-07 07:50:02","2026-05-07 11:24:11",{"title":3909,"description":48},{"loc":3963},"30072e6e8b386729","MarkTechPost","https:\u002F\u002Fwww.marktechpost.com\u002F2026\u002F05\u002F07\u002Fopenai-introduces-mrc-multipath-reliable-connection-a-new-open-networking-protocol-for-large-scale-ai-supercomputer-training-clusters\u002F","summaries\u002F30072e6e8b386729-mrc-openai-s-protocol-for-resilient-ai-training-ne-summary",[246,247,248],"OpenAI's MRC extends RoCE with multipath spraying, microsecond failure recovery via SRv6, and multi-plane designs to deliver predictable performance in 131k-GPU clusters, using 2\u002F3 fewer optics and 3\u002F5 fewer switches than traditional setups.",[],"XbDsma4E_5cuB3WLtPi6GgqSNlQtb2CdSK-eHkIrlrc",{"id":3977,"title":3978,"ai":3979,"body":3984,"categories":4012,"created_at":218,"date_modified":218,"description":48,"extension":219,"faq":218,"featured":220,"kicker_label":218,"meta":4013,"navigation":103,"path":4024,"published_at":4025,"question":218,"scraped_at":4026,"seo":4027,"sitemap":4028,"source_id":4029,"source_name":4030,"source_type":242,"source_url":4031,"stem":4032,"tags":4033,"thumbnail_url":218,"tldr":4034,"tweet":218,"unknown_tags":4035,"__hash__":4036},"summaries\u002Fsummaries\u002Ff78d6045a31221d2-mrc-enables-100k-gpu-clusters-with-resilient-multi-summary.md","MRC Enables 100k+ GPU Clusters with Resilient Multipath Networking",{"provider":7,"model":8,"input_tokens":3980,"output_tokens":3981,"processing_time_ms":3982,"cost_usd":3983},4244,1621,21683,0.00163665,{"type":14,"value":3985,"toc":4007},[3986,3990,3993,3997,4000,4004],[17,3987,3989],{"id":3988},"multipath-routing-fixes-core-bottlenecks-in-ai-training","Multipath Routing Fixes Core Bottlenecks in AI Training",[22,3991,3992],{},"MRC (Multipath Reliable Connection) eliminates congestion in AI supercomputers by distributing packets across hundreds of network paths simultaneously, rather than single paths. This delivers faster, more predictable GPU-to-GPU data transfers critical for training massive models. On failures—links, switches, or paths—MRC reroutes in microseconds, versus seconds or tens of seconds for standard 800 Gb\u002Fs fabrics. Result: Training jobs survive reboots and maintenance without stalls. OpenAI's multi-plane design connects over 100,000 GPUs using only two Ethernet switch tiers, slashing component count, power use, and costs compared to conventional three- or four-tier setups.",[17,3994,3996],{"id":3995},"proven-at-scale-on-frontier-supercomputers","Proven at Scale on Frontier Supercomputers",[22,3998,3999],{},"Deployed across OpenAI's largest NVIDIA GB200 clusters—including Oracle Cloud in Abilene, Texas, and Microsoft's Fairwater—MRC handled a real-world test during frontier model training for ChatGPT and Codex. Four tier-1 switches rebooted without coordinating with running jobs, proving zero-disruption resilience. This lets operators maintain networks mid-training, boosting uptime for trillion-parameter models where network stalls previously cost hours or days.",[17,4001,4003],{"id":4002},"open-standards-accelerate-adoption","Open Standards Accelerate Adoption",[22,4005,4006],{},"Specification released via Open Compute Project (OCP MRC 1.0), with contributions from AMD, Broadcom, Intel, Microsoft, and NVIDIA. Builders can implement now for Ethernet-based AI fabrics, avoiding proprietary lock-in while hitting supercomputer-scale performance.",{"title":48,"searchDepth":82,"depth":82,"links":4008},[4009,4010,4011],{"id":3988,"depth":82,"text":3989},{"id":3995,"depth":82,"text":3996},{"id":4002,"depth":82,"text":4003},[282],{"content_references":4014,"triage":4022},[4015,4017,4019],{"type":3889,"title":4016,"url":3891,"context":227},"Resilient AI Supercomputer Networking Using MRC and SRv6",{"type":224,"title":3886,"publisher":4018,"url":3887,"context":227},"Open Compute Project",{"type":224,"title":4020,"author":4021,"url":3959,"context":3956},"MRC Supercomputer Networking","OpenAI",{"relevance":88,"novelty":88,"quality":94,"actionability":82,"composite":3893,"reasoning":4023},"Category: AI & LLMs. The article discusses a new networking protocol that addresses bottlenecks in AI supercomputing, which is relevant to AI engineering. However, it lacks direct actionable insights for product builders on how to implement or leverage this technology in their own projects.","\u002Fsummaries\u002Ff78d6045a31221d2-mrc-enables-100k-gpu-clusters-with-resilient-multi-summary","2026-05-06 19:13:21","2026-05-07 11:24:04",{"title":3978,"description":48},{"loc":4024},"f78d6045a31221d2","The Decoder","https:\u002F\u002Fthe-decoder.com\u002Fopenai-built-a-networking-protocol-with-amd-broadcom-intel-microsoft-and-nvidia-to-fix-ai-supercomputer-bottlenecks\u002F","summaries\u002Ff78d6045a31221d2-mrc-enables-100k-gpu-clusters-with-resilient-multi-summary",[247,248,246],"OpenAI's MRC protocol spreads packets across hundreds of paths for microsecond failure recovery, connecting 100,000+ GPUs via just 2 switch tiers—cutting power, cost, and downtime in AI training supercomputers.",[],"LvMASfYTesYX0l3RENkA3FOBQpD3T6H-0KnDqYX6HvU",{"id":4038,"title":4039,"ai":4040,"body":4045,"categories":4079,"created_at":218,"date_modified":218,"description":48,"extension":219,"faq":218,"featured":220,"kicker_label":218,"meta":4080,"navigation":103,"path":4094,"published_at":4095,"question":218,"scraped_at":4096,"seo":4097,"sitemap":4098,"source_id":4099,"source_name":4100,"source_type":242,"source_url":4101,"stem":4102,"tags":4103,"thumbnail_url":218,"tldr":4104,"tweet":218,"unknown_tags":4105,"__hash__":4106},"summaries\u002Fsummaries\u002F6ee9b4f709da3a06-tpus-dominate-at-infrastructure-scale-over-per-chi-summary.md","TPUs Dominate at Infrastructure Scale Over Per-Chip GPU Wins",{"provider":7,"model":8,"input_tokens":4041,"output_tokens":4042,"processing_time_ms":4043,"cost_usd":4044},5399,1852,23082,0.00198315,{"type":14,"value":4046,"toc":4074},[4047,4051,4054,4057,4061,4064,4067,4071],[17,4048,4050],{"id":4049},"infrastructure-scaling-trumps-per-chip-performance","Infrastructure Scaling Trumps Per-Chip Performance",[22,4052,4053],{},"Google's TPU v8t for training and v8i for inference trail Nvidia's Rubin and AMD GPUs in raw per-chip compute and memory. However, evaluating at infrastructure level reveals TPUs' edge: Nvidia's NVL72 scales 72 Rubin GPUs per rack, while Google's 4x4x4 cube interconnects up to 9600 TPUs into a superpod delivering 121 exaFLOPS in FP4—surpassing Nvidia's 1152-GPU Rubin pod at 60 exaFLOPS FP4. Google's Virgo network further scales out to 134,000 chips, potentially reaching 1 million, minimizing network overhead via ICI and optical interconnects. This Lego-like modularity avoids the scaling cliffs Nvidia faces when stacking GPUs, where interconnect overhead erodes per-chip advantages.",[22,4055,4056],{},"Nvidia balances scale-out with InfiniBand for diverse customers (neo-clouds like CoreWeave, labs like OpenAI\u002FMeta, hyperscalers like Microsoft\u002FAmazon), prioritizing broad demand profiles. Google, serving internal apps like Gemini and Vertex AI plus external deals (Anthropic's $1B TPU commitment: 40% owned, 60% rented; Meta's multi-billion rental), optimizes purely for its high-volume needs without market fragmentation risks.",[17,4058,4060],{"id":4059},"workload-profiles-dictate-hardware-choices","Workload Profiles Dictate Hardware Choices",[22,4062,4063],{},"AI tasks bifurcate demands: training prioritizes network bandwidth over compute\u002Fmemory, benefiting TPU's topology. Inference splits further—prefill (pink line in SemiAnalysis chart) is compute\u002Fmemory-bound for KV cache parallelization; decode (white line) is bandwidth\u002Flatency-bound for autoregressive token streaming. TPU v8t\u002F8i bifurcation matches this: v8t for training's network focus, v8i for inference's varied needs. Virgo flattens network bottlenecks, challenging Nvidia's inference dominance.",[22,4065,4066],{},"Replicating Google's scaling on Nvidia chips risks inefficiency for its varied clientele, locking into a 'balanced diet' pod architecture over specialized superpods.",[17,4068,4070],{"id":4069},"explosive-demand-drives-economics","Explosive Demand Drives Economics",[22,4072,4073],{},"Epoch AI projects 450+ new pre-trained models by 2030, many exceeding GPT-5's ~66 septillion FLOPs (total math ops for weights). A 9600-TPU superpod could theoretically pretrain GPT-5-scale models in under 7 days at FP4 (realistically 3-4 weeks), but efficiency cliffs emerge from memory, bandwidth, or latency based on scale-up\u002Fout choices. Rising inference\u002Ftraining demand amplifies TPU economics: internal fab control ensures supply for massive token serving, positioning Google against Nvidia as workloads evolve toward bandwidth constraints.",{"title":48,"searchDepth":82,"depth":82,"links":4075},[4076,4077,4078],{"id":4049,"depth":82,"text":4050},{"id":4059,"depth":82,"text":4060},{"id":4069,"depth":82,"text":4070},[282],{"content_references":4081,"triage":4092},[4082,4085,4089],{"type":229,"title":4083,"url":4084,"context":3960},"Mammoth AI","http:\u002F\u002Fmammouth.ai",{"type":4086,"title":4087,"author":4088,"context":3956},"report","SemiAnalysis AI Demand Profiles Diagram","SemiAnalysis",{"type":4086,"title":4090,"author":4091,"context":3956},"Epoch AI Pre-Trained Models Projection","Epoch AI",{"relevance":88,"novelty":88,"quality":94,"actionability":82,"composite":3893,"reasoning":4093},"Category: AI & LLMs. The article discusses the performance of Google's TPUs compared to Nvidia GPUs, which is relevant to AI infrastructure but lacks direct actionable insights for product builders. While it provides some new perspectives on scaling AI workloads, it does not offer specific frameworks or techniques that the audience can implement.","\u002Fsummaries\u002F6ee9b4f709da3a06-tpus-dominate-at-infrastructure-scale-over-per-chi-summary","2026-04-30 02:16:18","2026-05-03 16:52:02",{"title":4039,"description":48},{"loc":4094},"a42442ea33b32f06","Caleb Writes Code","https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=b_KxiTPBIb0","summaries\u002F6ee9b4f709da3a06-tpus-dominate-at-infrastructure-scale-over-per-chi-summary",[246,248,247],"Google's TPU v8t (training) and v8i (inference) lag Nvidia GPUs per chip but deliver superior performance at scale—9600-chip superpods hit 121 exaFLOPS FP4—via cube topology and Virgo networking, optimizing for AI's bandwidth-heavy workloads.",[],"EDdnhIFUjAM7yxc1JlkafqMae9PUuMTF3sbgBpRfDo4"]