DataScale SN40L rack components and cabling

The DataScale SN40L rack internal components are delivered preconfigured and cabled.

You are not expected to interacted with DataScale rack component unless explicitly instructed by this document or the SambaNova support team.

You are expected to interact with the cables for network and power components that have to be connected to the datacenter to provide power and connectivity to the DataScale SN40L rack.

1. DataScale rack components

The following image and table identify the locations of the main components in the DataScale SN40L rack (rear facing).

DataScale SN40L rack (rear view)
Figure 1. DataScale SN40L rack (rear view)
Table 1. DataScale SN40L rack component identification
No. Component Description

1

SN40L-H-1 (system 1)

2

SN40L-H-1-XRDU0 (system 1)

3

SN40L-H-1-XRDU1 (system 1)

4

SN40L-H-1-XRDU2 (system 1)

5

SN40L-H-1-XRDU3 (system 1)

6

PDU 1 (power feed for system 1)

7

PDU 2 (redundant power feed for system 1)

8

PDU 3 (power feed for system 2)

9

PDU 4 (redundant power feed for system 2)

10

SN40L-H-2 (system 2)

11

SN40L-H-2-XRDU0 (system 2)

12

SN40L-H-2-XRDU1 (system 2)

13

SN40L-H-2-XRDU2 (system 2)

14

SN40L-H-2-XRDU3 (system 2)

15

Juniper QFX5130 Ethernet high-bandwidth data switch

16

Juniper EX-series access switch

17

Lantronix serial console server

2. Cabling system components

The DataScale SN40L rack comes preconfigured and cabled, with each system connected to the appropriate switches and switch ports. The following images and tables identify the components that are either connected to switches or to other system components.

This section is for information only. Do not interact with these components unless instructed to do so by the SambaNova support team.
SN40L-H (rear view)
Figure 2. SN40L-H (rear view)
Table 2. SN40L-H components (rear view)
No. Component Connector/Cable Type

1

Two power inlets

C13 to C14 power cord

2

SN40L-H Net0 1GbE LAN

RJ45, Cat6

3

SN40L-H 10/100/1000 management LAN (labeled M)

RJ45, Cat6

4

DB9 serial connection

Rollover DB9F to RJ45 console cable

5

Host interface card (HIC)

QSFP-DD cable

6

High-bandwidth network ports

QSFP56

SN40L-2 (rear view)
Figure 3. SN40L-2 (rear view)
Table 3. SN40L-2 components (rear view)
No. Component Connector/Cable Type

1

Host HIC connections

QSFP-DD cable

2

Two power inlets

C19 to C20 power cord

3

SN40L-2 COM serial port (COM)

RJ45, Cat6

4

SN40L-2 10/100/1000 management LAN (MGMT)

RJ45, Cat6

5

High-bandwidth network port connections

QSFP56

3. Serial console server cabling

The Latronix serial console server is installed and cabled in the factory.

This section is for information only. Do not interact with these components unless instructed to do so by the SambaNova support team.

This component, located in RU42, occupies the front portion of the same rack unit as the Juniper EX series access switch. The serial console server uses standard Ethernet RJ45 cables and rollover DB9F to RJ45 console cables for the DataScale SN40L-H host module.

A dedicated non-tagged VLAN network drop, which is on the access/management network, is required for the serial console server. This connection goes to the serial console server management port indicated by the number 1 in Lantronix serial console server (rear view). Having a dedicated serial console server network drop ensures that there is still network connectivity to the serial console server if anything happens to the Juniper EX series access switch.

Ports for the serial console server are not visible or accessible to the user because of the physical location of the unit in front of the access switch. This component is primarily intended for SambaNova service operations. The cabling is effectively hidden from the user, but the network port labeled 1 is brought out to one of the dongles labeled SCS Dedicated network uplink.
Lantronix Serial Console Server (rear view)
Figure 4. Lantronix serial console server (rear view)
Table 4. Lantronix serial console server components (rear view)
No. Component Connector/Cable Type

1

10/100/1000 management Network (labeled 1)

RJ45, Cat6 or higher

2

Serial RJ45 connections (ports 1–32)

RJ45, Ethernet cable

3

Two power inlets

C13 to C14 power cord

Cabling requirements from the Lantronix serial console server to the other DataScale SN40L rack components are listed in the following table.

Table 5. Serial console server connections
From Serial Console Server Port To Component Component Location Component Port

1

Juniper EX-series
access switch

RU42

CON2

2

Juniper QFX5130 Ethernet high-bandwidth data switch

RU41

CON (Juniper)

3

PDU 1

RU16/RU17

RJ45 just left of USB labeled 10101

4

PDU 2

RU18/RU19

RJ45 just left of USB labeled 10101

5

PDU 3

RU20/RU21

RJ45 just left of USB labeled 10101

6

PDU 4

RU22/RU23

RJ45 just left of USB labeled 10101

7-11

-

-

-

12

SN40L-H-2-XRDU3

RU33/RU34

COM

13

SN40L-H-2-XRDU2

RU31/RU32

COM

14

SN40L-H-2-XRDU1

RU29/RU30

COM

15

SN40L-H-2-XRDU0

RU27/RU28

COM

16

SN40L-H-2

RU25/RU26

DB9 male PCI card

17-21

-

-

-

22

SN40L-H-1-XRDU3

RU13/RU14

COM

23

SN40L-H-1-XRDU2

RU11/RU12

COM

24

SN40L-H-1-XRDU1

RU9/RU10

COM

25

SN40L-H-1-XRDU0

RU7/RU8

COM

26

SN40L-H-1

RU5/RU6

DB9 male PCI card

27-32

-

-

-

4. Access switch cabling

The Juniper EX series switch that is used for the access networks is installed and cabled in the factory.

You are expected to interact only with port 47 of the access switch to connect an uplink to their network. The rest of this section is for information only. Do not interact with the other components unless instructed to do so by the SambaNova support team.

This component is located in RU42 at the rear of the rack and is shared in the same rack unit as the Lantronix serial console server. This switch uses standard Cat6 Ethernet RJ45 cables. Customers can further segregate this switch using port-based VLANs to separate access to the management interfaces of the DataScale SN40L rack components and the interface of the SN40L-Hs operating system(OS). The customer is responsible for configuring this added network segregation. Access network port cabling shows what device is connected to which port, and shows available ports that can be used if an additional network drop is required for the separate network. If requested, SambaNova staff can configure this port separation during initial deployment.

You must use a separate network drop for each network when configuring the access switch to separate the management interfaces and the SN40L-H OS interfaces with port-based VLANs. If the switch is reset to factory default for any reason, the VLAN configuration is lost. This could result in spanning-tree protocols disabling one or both of the networks. By default, the access switch is configured for Rapid Spanning Tree Protocol (RSTP).
Juniper EX series Access Switch (front view)
Figure 5. Juniper EX series access switch (front view)
Juniper EX series Access Switch (rear view)
Figure 6. Juniper EX series access switch (rear view)
Table 6. Juniper EX series access switch components
No. Component Connector/Cable Type

1

1GbE ports (Ports 0–47)

RJ45, Cat6

2

RJ45 console port

RJ45, Cat6

3

Network management port for em0

RJ45, Cat6

4

Two power inlets

C13 to C14 power cord

Cabling requirements from the Juniper EX series switch for the access network to the other DataScale SN40L rack components are listed in Access network port cabling.

The network management port is brought out to a dongle at the rear of the rack to allow customers to connect a dedicated cable to the management port of the Juniper EX series switch for recovery purposes. This dongle is labeled ACCSW Dedicated Management Uplink.

Table 7. Access network port cabling
From Access
Switch Port
To Component Component Location Component Port

0-1

-

-

-

2

Juniper QFX5130 Ethernet high-bandwidth data switch

RU41

MGMT

3

SN40L-H-1-XRDU0 (system 1)

RU7/RU8

MGMT

4

SN40L-H-1-XRDU1 (system 1)

RU9/RU10

MGMT

5

SN40L-H-1-XRDU2 (system 1)

RU11/RU12

MGMT

6

SN40L-H-1-XRDU3 (system 1)

RU13/RU14

MGMT

7-10

-

-

-

11

SN40L-H-2-XRDU0 (system 2)

RU27/RU28

MGMT

8

SN40L-H-2-XRDU1 (system 2)

RU29/RU30

MGMT

9

SN40L-H-2-XRDU2 (system 2)

RU31/RU32

MGMT

10

SN40L-H-2-XRDU3 (system 2)

RU32/RU34

MGMT

15-25

-

-

-

26

PDU 1

RU16/RU17

RJ45 labeled 1

27

PDU 2

RU18/RU19

RJ45 labeled 1

28

PDU 3

RU20/RU21

RJ45 Labeled 1

29

PDU 4

RU22/RU23

RJ45 Labeled 1

30

SN40L-H-1

RU5/RU6

Labeled M

31

SN40L-H-2

RU25/RU26

Labeled M

32-39

-

-

-

40

Management Network Uplink
(Untagged VLAN)

-

-

41

Access Network Uplink
(Untagged VLAN)

-

-

42

-

-

-

43

SN40L-H-1

RU5/RU6

Left port Net0

44

SN40L-H-2

RU25/RU26

Left port Net0

45-46

-

-

-

47

Access and Management Network Uplink
(with Tagged VLANs)

Plug your uplink into this port.

-

-

5. High-bandwidth data switch cabling

The DataScale SN40L rack is configured with a Juniper QFX5130 400GbE Ethernet switch. The connections to the SN40L-2 and SN40L-H provide 200GbE connectivity to the data switch using 400GbE to 2 x 200GbE splitter cables.

You are expected to interact only with port 27-30 or port 31 to connect an uplink to their network. The rest of this section is for information only. Do not interact with other components unless instructed to do so by the SambaNova support team.

5.1. High-bandwidth Ethernet switch

The Juniper QFX5130 switch, which is the standard high-bandwidth data switch in the DataScale SN40L rack, is installed and cabled in the factory. This switch is located in RU41.

Juniper QFX5130 High-Bandwidth Ethernet Data Switch (Front View)
Figure 7. Juniper QFX5130 high-bandwidth Ethernet data switch (front view)
Juniper QFX5130 High-Bandwidth Ethernet Data Switch (Rear View)
Figure 8. Juniper QFX5130 high-bandwidth Ethernet data switch (rear view)
Table 8. Juniper QFX5130 high-bandwidth Ethernet data switch components
No. Component Connector/Cable Type

1

400GbE ports (ports 0–31)

QSFP-DD

2

Network management port for em0

RJ45, Cat6

3

RJ45 console port connection (CON)

RJ45, Ethernet serial cable

4

Two power inlets

C13 to C14 power cord

5.2. SN40L-H and SN40L-2 high-bandwidth network cabling

The high-bandwidth cabling for the SN40L-Hs and SN40L-2 systems is installed in the factory before delivery.

The cabling used for this is high-bandwidth 400GbE to a two x 200-GbE Direct Attach Copper (DAC) splitter cable. For the SN40L-2 the figure below, Slot 1 is also referred to as PCIe Riser 1 - bottom port.

SN40L-H high-bandwidth network card port assignment
Figure 9. SN40L-H high-bandwidth network card port assignment
SN40L-2 high-bandwidth network card port assignment
Figure 10. SN40L-2 high-bandwidth network card port assignment

Cabling requirements from the Juniper QFX5130 Ethernet switch for the high-bandwidth data network to the other DataScale SN40L rack components are listed in the following table.

Table 9. high-bandwidth data network port cabling (Ethernet, default)
From Data Network Switch Port To Component Component Location Component Port

0:0

SN40L-H-1-XRDU0 (system 1)

RU7/RU8

PCIe slot 1 - Port 0

0:1

SN40L-H-1-XRDU1 (system 1)

RU9/RU10

PCIe slot 1 - Port 0

1:0

SN40L-H-1-XRDU2 (system 1)

RU11/RU12

PCIe slot 1 - Port 0

1:1

SN40L-H-1-XRDU3 (system 1)

RU13/RU14

PCIe slot 1 - Port 0

2:0

SN40L-H-2-XRDU0 (system 2)

RU27/RU28

PCIe slot 1 - Port 0

2:1

SN40L-H-2-XRDU1 (system 2)

RU29/RU30

PCIe slot 1 - Port 0

3:0

SN40L-H-2-XRDU2 (system 2)

RU31/RU32

PCIe slot 1 - Port 0

3:1

SN40L-H-1-XRDU3 (system 2)

RU33/RU34

PCIe slot 1 - Port 0

16:0

SN40L-H-1 (system 1)

RU5/RU6

PCIe slot 3 - Port 0

16:1

SN40L-H-2 (system 2)

RU25/RU26

PCIe slot 3 - Port 0

27–30

LAG uplinks to customer data network

-

-

31

Non-LAG uplink to customer data network

-

-

6. Power cabling of the DataScale SN40L rack

DataScale SN40L racks are delivered preinstalled with power cables from the factory.

This section is for information only. Do not interact with these components unless instructed to do so by the SambaNova support team.

Four power distribution units (PDUs) are installed in the DataScale SN40L rack. The PDUs provide redundant power in case of a device PSU failure, a single PDU failure, or a datacenter power feed failure. They are located on the left side and the right side of the rack when you are facing the rear of the rack. The following table is a reference to the PDU identification and location within the rack.

Table 10. PDU location in DataScale SN40L rack
PDU Identification Location in rack (facing rack rear)

PDU1

RU16/RU17

PDU2

RU18/RU19

PDU3

RU20/RU21

PDU4

RU22/RU23

Port population of the PDUs varies depending on the number of systems in the DataScale SN40L rack, but the same outlets are always used for the same systems regardless of rack configuration. [table15] shows all the required connections for a two node rack. The information in the table also applies to a rack with fewer populated systems.

Do NOT use unused ports for any purpose. To prevent overloading of breaker circuits in the PDU and unexpected outages use only the designated ports.
Table 11. PDU to component power supply mapping
From Rack Unit From Component To PDU1 Port To PDU2 Port To PDU3 Port To PDU4 Port

42

Lantronix console server PSU1 (left PSU when facing rear of Lantronix)

1

-

-

-

42

Lantronix console server PSU2 (right PSU when facing rear of Lantronix)

-

1

-

-

42

Juniper EX-series access switch PSU 1 (Left PSU when facing rear of switch)

2

-

-

-

42

Juniper EX-series access switch PSU 2 (right PSU when facing rear of switch)

-

2

-

-

41

Juniper QFX5130 data switch Eth PSU 1 (left PSU when facing rear of switch)

13

-

-

-

41

Juniper QFX5130 data switch Eth PSU 1 (right PSU when facing rear of switch)

-

13

-

-

34

SN40L-H-2-XRDU3 top PSU (system 2)

-

-

24

-

33

SN40L-H-2-XRDU3 bottom PSU (system 2)

-

-

-

24

32

SN40L-H-2-XRDU2 top PSU (system-2)

-

-

22

-

31

SN40L-H-2-XRDU2 bottom PSU (system-2)

-

-

-

22

30

SN40L-H-2-XRDU1 top PSU (system 2)

-

-

19

-

29

SN40L-H-2-XRDU1 bottom PSU (system 2)

-

-

-

19

28

SN40L-H-2-XRDU0 top PSU (system 2)

-

-

7

-

27

SN40L-H-2-XRDU0 bottom (system 2)

-

-

-

7

25

SN40L-H-2 left PSU (system 2)

-

-

3

-

25

SN40L-H-2 right PSU (system 2)

-

-

-

3

14

SN40L-H-1-XRDU3 top PSU (system 1)

10

-

-

-

13

SN40L-H-1-XRDU3 bottom PSU (system 1)

-

10

-

-

12

SN40L-H-1-XRDU2 top PSU (system 1)

12

-

-

-

11

SN40L-H-1-XRDU2 bottom PSU (system 1)

-

12

-

-

10

SN40L-H-1-XRDU1 top PSU (system 1)

22

-

-

-

9

SN40L-H-1-XRDU1 bottom PSU (system 1)

-

22

-

-

8

SN40L-H-1-XRDU0 top PSU (system 1)

24

-

-

-

7

SN40L-H-1-XRDU0 bottom PSU (system 1)

-

24

-

-

5

SN40L-H-1 left PSU (system 1)

14

-

-

-

5

SN40L-H-1 right PSU (system 1)

-

14

-

-