smalllife // packet & bench

Where six idle watts went: PCIe ASPM L1.2 and copper SFP+ modules

2025-11-02 · hardware, power, homelab

Six watts sounds trivial until you multiply it across a year (~52 kWh) and notice what those six watts actually become inside a fanless aluminum chassis: another 11 °C on the NVMe controller every summer afternoon.

1. The 10GBASE-T SFP+ tax

My switch had two SFP+ cages. Plugging a generic 10GBASE-T RJ45 module into cage 9 raised wall power on sw01 from 3.8 W to 6.2 W with the cable unplugged on the far end, and 6.9 W at 2.5G link-up. Swapping that short 1.5-meter run for a passive twinax DAC dropped total switch power right back to 3.9 W at 10 Gbps full link.

2. Package C-states blocked by one PCIe endpoint

On gw01 (Alder Lake-N N100), powertop showed individual cores spending 96% of idle time in C10 while the SoC package never went deeper than PC2, idling at 9.4 W at the wall.

$ lspci -vv | awk '/^[0-9a-f]{2}:/{dev=$0} /LnkCtl:/{print dev "\n  " $0}'
01:00.0 Non-Volatile memory controller: ...
  LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
02:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-V (rev 04)
  LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+

BIOS had PCIe ASPM set to Auto, which left all four i226-V NICs with ASPM disabled and EEE turned off. Enabling ASPM L1 + L1.2 substates in firmware setup (rather than forcing pcie_aspm.policy=powersupersave on untested endpoints) let the package drop into PC8/PC10:

ConfigurationDeepest Pkg StateWall ACCase Temp
BIOS defaults + RJ45 SFP+PC2 (78%)11.9 W46.5 °C
ASPM L1.2 enabled on i226-V + NVMePC8 (84%)6.1 W37.0 °C
+ EEE off on WAN, on for idle portsPC10 (61%)5.8 W35.8 °C

Note: if i226-V drops link briefly with older switches under EEE, leave ASPM L1 enabled (PCIe bus side) and disable only 802.3az EEE via ethtool --set-eee enp1s0 eee off; almost all of the package C-state win comes from PCIe L1.2, not PHY EEE.