Background

I would like to clarify the behavior and intention behind the pcieport driver actions during device removal.

First, I reviewed the PCI-PM and PCIe Gen6 Base Specifications, focusing on the sections related to power management (PCI-PM). According to the PCIe specifications:

• Ch.5.3.2.1 Entry into the L1 State
• Ch.5.4.1.3.1 ASPM Entry into the L1 State

I concluded two rules:

• A downstream port cannot initiate L1.
• A downstream port can only respond to an L1 entry request; it cannot initiate the request.

System Configuration:

• PCIe Root Port at 00:03.0
• PCIe Endpoint (EP) at 01:00.0
• Linux Kernel Version: 6.3.0 (x86_64) on Ubuntu 22.04.3 LTS

I have added printk statements in dmesg for various files under drivers/pci/*.c and used my internal tools to trace the driver behavior (CfgRd/CfgWr) for PCIe Gen6 Type1 & Type0 devices.

QUESTION

When I run the command “echo 1 | sudo tee /sys/bus/pci/devices/0000:01:00.0/remove” to remove the endpoint, I observe the following behavior:

• CfgWr to PMCSR of Downstream Port to D3_hot
• The pcieport driver writes the PMCSR of the downstream port to D3_hot (config_write: bdf: 00:03.0 addr: 78 data: 10b) after the device 0000:01:00.0 is removed. Is there a specific reason for this action?
• The downstream port keeps generating PME? as indicated by the following function calls. I wonder why does the driver exhibit this behavior?

pci_pme_list_scan
pci_pme_wakeup
pci_check_pme_status
pci_read_config_word
pci_pme_list_scan
pci_pme_wakeup
pci_check_pme_status
pci_read_config_word
pci_pme_list_scan
pci_pme_wakeup
pci_check_pme_status
pci_read_config_word
... (repeating until rescan pci)

When I run “echo 1 | sudo tee /sys/bus/pci/rescan”, the pcieport driver first writes the downstream port to D0, and then the sysfs adds the endpoint 0000:01:00.0 back. This process does not seem related to the LTSSM initiating L1.

It appears that the driver behavior is to simply change the downstream port’s D-state without involving an L1 request.

What is the reasoning behind this behavior?

Logs

Here is a summarized trace of the functions called during the “remove” process (I may not be 100% sure of the hierarchy sequence, but I am certain these functions are involved):

######## echo 1 | sudo tee /sys/bus/pci/devices/0000:01:00.0/remove ########
drivers/pci/pci-sysfs.c:485 (remove_store)
        drivers/pci/remove.c:123 (pci_stop_and_remove_bus_device_locked)
                drivers/pci/probe.c:3300 (pci_lock_rescan_remove)
                
                drivers/pci/remove.c:116 (pci_stop_and_remove_bus_device)
                        drivers/pci/remove.c:67 (pci_stop_bus_device)
                                drivers/pci/remove.c:18 (pci_stop_dev)
                                        drivers/pci/pci.c:2462 (pci_pme_active)
                                                drivers/pci/pci.c:2416 (__pci_pme_active)
                                                        drivers/pci/access.c:545 (pci_read_config_word)  # config_read: bdf: 01:00:0 addr: 48 data: 8
                                                        drivers/pci/access.c:574 (pci_write_config_word) # config_write: bdf: 01:00:0 addr: 48 data: 8008
                            
                            # ...
                            # drivers/base/dd.c:1280 (device_release_driver)
                            # drivers/pci/proc.c:448 (pci_proc_detach_device)
                            # drivers/pci/pci-sysfs.c:1510 (pci_remove_sysfs_dev_files)
                            # drivers/pci/pci-sysfs.c:1149 (pci_remove_resource_files)
                          # ...
                        
                        drivers/pci/remove.c:87 (pci_remove_bus_device)
                                drivers/pci/remove.c:32 (pci_destroy_dev)
                                
                                        drivers/pci/vgaarb.c:1505 (pci_notify)
                                                drivers/pci/vgaarb.c:844 (vga_arbiter_del_pci_device)
                                                # dmesg shows "bus: 'pci': remove device 0000:01:00.0"
                    
                    
                        # ...
                        # drivers/pci/pci-driver.c:1361 (pci_pm_runtime_idle)
                                # drivers/pci/pcie/portdrv.c:641 (pcie_port_runtime_idle)
                                # drivers/pci/pci-driver.c:1274 (pci_pm_runtime_suspend)
                                
                                # drivers/pci/pcie/portdrv.c:633 (pcie_port_runtime_suspend)
                                        # drivers/pci/pcie/portdrv.c:427 (pcie_port_device_runtime_suspend)
                                        
                                                #### drivers/pci/pci.c:1646 (pci_save_state) #### (do many config_read...)
                                                pci_save_pcie_state(dev);
                                                pci_save_pcix_state(dev);
                                                pci_save_ltr_state(dev);
                                                pci_save_dpc_state(dev);
                                                pci_save_aer_state(dev);
                                                pci_save_ptm_state(dev);
                                                return pci_save_vc_state(dev);
                                                
                                                #### drivers/pci/pci.c:1493 (pci_save_pcie_state) #### (do many config_read...)
                                                pcie_capability_read_word(dev, PCI_EXP_DEVCTL, &cap[i++]);
                                                pcie_capability_read_word(dev, PCI_EXP_LNKCTL, &cap[i++]);
                                                pcie_capability_read_word(dev, PCI_EXP_SLTCTL, &cap[i++]);
                                                pcie_capability_read_word(dev, PCI_EXP_RTCTL,  &cap[i++]);
                                                pcie_capability_read_word(dev, PCI_EXP_DEVCTL2, &cap[i++]);
                                                pcie_capability_read_word(dev, PCI_EXP_LNKCTL2, &cap[i++]);
                                                pcie_capability_read_word(dev, PCI_EXP_SLTCTL2, &cap[i++]);
                                                return 0;
                                                
                                # drivers/pci/pci.c:2746 (pci_finish_runtime_suspend)
                                        # drivers/pci/pci.c:2640 (pci_target_state)
                                        
                                # ...
                                # ...
                                # ...
                                
                                # drivers/pci/pci.c:2416 (__pci_pme_active)
                                        # drivers/pci/access.c:545 (pci_read_config_word)  # config_read: bdf: 00:03:0 addr: 78 data: 8 size: 2 bytes
                                        # drivers/pci/access.c:574 (pci_write_config_word) # config_write: bdf: 00:03:0 addr: 78 data: 8108 size: 2 bytes

                                # drivers/pci/pci.c:1412 (pci_set_power_state)
                                        # drivers/pci/pci.c:1339 (pci_set_low_power_state)
                                                # drivers/pci/access.c:545 (pci_read_config_word)  # config_read: bdf: 00:03:0 addr: 78 data: 108 size: 2 bytes
                                                # drivers/pci/access.c:574 (pci_write_config_word) # config_write: bdf: 00:03:0 addr: 78 data: 10b size: 2 bytes
                                                
                                        # drivers/pci/pci.c:68 (pci_dev_d3_sleep)
                
                # ...
                drivers/pci/probe.c:3306 (pci_unlock_rescan_remove)
                
        # REPEAT...
        [ 2812.018124] [JAY-DBG-PCI]drivers/pci/access.c:545 (pci_read_config_word) # config_read: bdf: 00:03:0 addr: 78 data: 10b size: 2 bytes
        [ 2812.018563] [JAY-DBG-PCI]drivers/pci/pci.c:1120 (pci_platform_power_transition)
        [ 2812.018565] [JAY-DBG-PCI]drivers/pci/pci.c:1018 (platform_pci_set_power_state)
        [ 2812.018566] [JAY-DBG-PCI]drivers/pci/pci-acpi.c:1047 (acpi_pci_set_power_state)
        [ 2813.034196] [JAY-DBG-PCI]drivers/pci/pci.c:2380 (pci_pme_list_scan)
        [ 2813.034238] [JAY-DBG-PCI]drivers/pci/pci.c:2343 (pci_pme_wakeup)
        [ 2813.034238] [JAY-DBG-PCI]drivers/pci/pci.c:2308 (pci_check_pme_status)
        [ 2813.034239] [JAY-DBG-PCI]drivers/pci/access.c:545 (pci_read_config_word) # config_read: bdf: 00:03:0 addr: 78 data: 10b size: 2 bytes
        [ 2814.058170] [JAY-DBG-PCI]drivers/pci/pci.c:2380 (pci_pme_list_scan)
        [ 2814.058177] [JAY-DBG-PCI]drivers/pci/pci.c:2343 (pci_pme_wakeup)
        [ 2814.058177] [JAY-DBG-PCI]drivers/pci/pci.c:2308 (pci_check_pme_status)
        [ 2814.058178] [JAY-DBG-PCI]drivers/pci/access.c:545 (pci_read_config_word) # config_read: bdf: 00:00:0 addr: 78 data: 10b size: 2 bytes
        [ 2815.082026] [JAY-DBG-PCI]drivers/pci/pci.c:2380 (pci_pme_list_scan)
        [ 2815.082031] [JAY-DBG-PCI]drivers/pci/pci.c:2343 (pci_pme_wakeup)
        [ 2815.082032] [JAY-DBG-PCI]drivers/pci/pci.c:2308 (pci_check_pme_status)
        [ 2815.082033] [JAY-DBG-PCI]drivers/pci/access.c:545 (pci_read_config_word) # config_read: bdf: 00:03:0 addr: 78 data: 10b size: 2 bytes
        [ 2816.106322] [JAY-DBG-PCI]drivers/pci/pci.c:2380 (pci_pme_list_scan)
        [ 2816.106335] [JAY-DBG-PCI]drivers/pci/pci.c:2343 (pci_pme_wakeup)
        [ 2816.106337] [JAY-DBG-PCI]drivers/pci/pci.c:2308 (pci_check_pme_status)
        [ 2816.106339] [JAY-DBG-PCI]drivers/pci/access.c:545 (pci_read_config_word) # config_read: bdf: 00:03:0 addr: 78 data: 10b size: 2 bytes
        [ 2817.130301] [JAY-DBG-PCI]drivers/pci/pci.c:2380 (pci_pme_list_scan)
        [ 2817.130306] [JAY-DBG-PCI]drivers/pci/pci.c:2343 (pci_pme_wakeup)
        [ 2817.130307] [JAY-DBG-PCI]drivers/pci/pci.c:2308 (pci_check_pme_status)
        [ 2817.130307] [JAY-DBG-PCI]drivers/pci/access.c:545 (pci_read_config_word) # config_read: bdf: 00:03:0 addr: 78 data: 10b size: 2 bytes
        [ 2818.154190] [JAY-DBG-PCI]drivers/pci/pci.c:2380 (pci_pme_list_scan)
        [ 2818.154196] [JAY-DBG-PCI]drivers/pci/pci.c:2343 (pci_pme_wakeup)
        [ 2818.154197] [JAY-DBG-PCI]drivers/pci/pci.c:2308 (pci_check_pme_status)
        [ 2818.154198] [JAY-DBG-PCI]drivers/pci/access.c:545 (pci_read_config_word) # config_read: bdf: 00:03:0 addr: 78 data: 10b size: 2 bytes
        [ 2819.178095] [JAY-DBG-PCI]drivers/pci/pci.c:2380 (pci_pme_list_scan)
        [ 2819.178099] [JAY-DBG-PCI]drivers/pci/pci.c:2343 (pci_pme_wakeup)
        [ 2819.178100] [JAY-DBG-PCI]drivers/pci/pci.c:2308 (pci_check_pme_status)
        [ 2819.178101] [JAY-DBG-PCI]drivers/pci/access.c:545 (pci_read_config_word) # config_read: bdf: 00:03:0 addr: 78 data: 10b size: 2 bytes
        [ 2820.202248] [JAY-DBG-PCI]drivers/pci/pci.c:2380 (pci_pme_list_scan)
        [ 2820.202252] [JAY-DBG-PCI]drivers/pci/pci.c:2343 (pci_pme_wakeup)
        [ 2820.202252] [JAY-DBG-PCI]drivers/pci/pci.c:2308 (pci_check_pme_status)
        [ 2820.202253] [JAY-DBG-PCI]drivers/pci/access.c:545 (pci_read_config_word) # config_read: bdf: 00:03:0 addr: 78 data: 10b size: 2 bytes
        
        
        
The following is the detail lspci result for my PCIe devices

qemu@qemu:~$ sudo lspci -kvvv
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
    Subsystem: Red Hat, Inc. QEMU Virtual Machine
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0


00:03.0 PCI bridge: Vadatech Inc. Device 1234 (rev 17) (prog-if 00 [Normal decode])
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0
    Interrupt: pin A routed to IRQ 23
    Region 0: Memory at 800000000 (64-bit, prefetchable) [size=64K]
    Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
    I/O behind bridge: 0000f000-00000fff [disabled]
    Memory behind bridge: c0900000-c09fffff [size=1M]
    Prefetchable memory behind bridge: 00000000c0800000-00000000c08fffff [size=1M]
    Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
    Expansion ROM at 80040000 [disabled] [size=32K]
    BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16- MAbort- >Reset- FastB2B-
        PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
    Capabilities: [74] Power Management version 3
        Flags: PMEClk- DSI- D1- D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [98] MSI: Enable- Count=1/32 Maskable+ 64bit+
        Address: 00000000fee00000  Data: 00ef
        Masking: 00000000  Pending: 00000000
    Capabilities: [b8] Express (v2) Root Port (Slot+), MSI 1a
        DevCap: MaxPayload 2048 bytes, PhantFunc 0
            ExtTag+ RBE+
        DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 128 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
        LnkCap: Port #0, Speed unknown, Width x8, ASPM L1, Exit Latency L1 unlimited
            ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
        LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed unknown (ok), Width x8 (ok)
            TrErr- Train- SlotClk- DLActive+ BWMgmt+ ABWMgmt-
        SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
            Slot #0, PowerLimit 0.000W; Interlock- NoCompl-
        SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
            Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
        SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
            Changed: MRL- PresDet- LinkState-
        RootCap: CRSVisible+
        RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible+
        RootSta: PME ReqID 0000, PMEStatus- PMEPending-
        DevCap2: Completion Timeout: Range BC, TimeoutDis+ NROPrPrP- LTR-
             10BitTagComp+ 10BitTagReq+ OBFF Via message/WAKE#, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 3
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- LN System CLS 128byte cachelines, TPHComp+ ExtTPHComp- ARIFwd+
             AtomicOpsCap: Routing+ 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled, ARIFwd-
             AtomicOpsCtl: ReqEn- EgressBlck-
        LnkCap2: Supported Link Speeds: RsvdP, Crosslink- Retimer+ 2Retimers+ DRS+
        LnkCtl2: Target Link Speed: Unknown, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
             EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported, DRS+
             DownstreamComp: Reserved
    Capabilities: [100 v1] Extended Capability ID 0x2f
    Capabilities: [154 v3] Advanced Error Reporting
        UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
        CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
        RootCmd: CERptEn+ NFERptEn+ FERptEn+
        RootSta: CERcvd- MultCERcvd- UERcvd- MultUERcvd-
             FirstFatal- NonFatalMsg- FatalMsg- IntMsg 0
        ErrorSrc: ERR_COR: 0000 ERR_FATAL/NONFATAL: 0000
    Capabilities: [270 v1] Virtual Channel
        Caps:   LPEVC=0 RefClk=100ns PATEntryBits=8
        Arb:    Fixed- WRR32- WRR64- WRR128-
        Ctrl:   ArbSelect=Fixed
        Status: InProgress-
        VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
            Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
            Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
            Status: NegoPending- InProgress-
    Capabilities: [4ec v1] Root Complex Link
        Desc:   PortNumber=00 ComponentID=01 EltType=Config
        Link0:  Desc:   TargetPort=00 TargetComponent=00 AssocRCRB- LinkType=Config LinkValid-
            Addr:   00:00.0  CfgSpace=0000000000000000
    Capabilities: [738 v1] Root Complex Internal Link <?>
    Capabilities: [75c v1] Power Budgeting <?>
    Capabilities: [770 v1] Secondary PCI Express
        LnkCtl3: LnkEquIntrruptEn- PerformEqu-
        LaneErrStat: 0
    Capabilities: [794 v1] L1 PM Substates
        L1SubCap: PCI-PM_L1.2- PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1- L1_PM_Substates+
              PortCommonModeRestoreTime=150us PortTPowerOnTime=10us
        L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
               T_CommonMode=150us LTR1.2_Threshold=0ns
        L1SubCtl2: T_PwrOn=10us
    Capabilities: [7b4 v1] Physical Layer 16.0 GT/s <?>
    Capabilities: [7dc v1] Lane Margining at the Receiver <?>
    Capabilities: [83c v1] Data Link Feature <?>
    Capabilities: [874 v1] Extended Capability ID 0x2a
    Capabilities: [8b0 v1] Extended Capability ID 0x31
    Capabilities: [8d0 v1] Extended Capability ID 0x32
    Capabilities: [924 v1] Extended Capability ID 0x33
    Capabilities: [9d0 v1] Extended Capability ID 0x34
    Kernel driver in use: pcieport


01:00.0 Mass storage controller: Device 18ef:1234 (rev 49)
    Subsystem: Seeq Technology, Inc. Device 58e0
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0
    Interrupt: pin A routed to IRQ 11
    Region 0: Memory at c0800000 (64-bit, prefetchable) [size=1M]
    Region 2: Memory at c0900000 (64-bit, non-prefetchable) [size=1M]
    Expansion ROM at <ignored> [disabled]
    Capabilities: [44] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [50] MSI: Enable- Count=1/2 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [60] MSI-X: Enable- Count=19 Masked-
        Vector table: BAR=0 offset=00000000
        PBA: BAR=0 offset=00000130
    Capabilities: [6c] Express (v2) Endpoint, MSI 0f
        DevCap: MaxPayload 1024 bytes, PhantFunc 7, Latency L0s unlimited, L1 <64us
            ExtTag+ AttnBtn- AttnInd+ PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
        DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
            MaxPayload 128 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
        LnkCap: Port #0, Speed unknown, Width x8, ASPM L1, Exit Latency L1 unlimited
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed unknown (ok), Width x8 (ok)
            TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
             10BitTagComp+ 10BitTagReq- OBFF Via message/WAKE#, ExtFmt+ EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS- TPHComp+ ExtTPHComp+
             AtomicOpsCap: 32bit+ 64bit+ 128bitCAS+
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
             AtomicOpsCtl: ReqEn-
        LnkCap2: Supported Link Speeds: RsvdP, Crosslink- Retimer+ 2Retimers+ DRS+
        LnkCtl2: Target Link Speed: Unknown, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
             EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported
    Capabilities: [100 v1] Extended Capability ID 0x2f
    Capabilities: [1e8 v3] Advanced Error Reporting
        UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
        CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
    Capabilities: [240 v1] Virtual Channel
        Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
        Arb:    Fixed- WRR32- WRR64- WRR128-
        Ctrl:   ArbSelect=Fixed
        Status: InProgress-
        VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
            Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
            Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
            Status: NegoPending- InProgress-
    Capabilities: [31c v1] Device Serial Number ab-cd-ef-01-23-45-67-89
    Capabilities: [32c v1] Latency Tolerance Reporting
        Max snoop latency: 0ns
        Max no snoop latency: 0ns
    Capabilities: [344 v1] Secondary PCI Express
        LnkCtl3: LnkEquIntrruptEn- PerformEqu-
        LaneErrStat: 0
    Capabilities: [3e0 v1] Address Translation Service (ATS)
        ATSCap: Invalidate Queue Depth: 1b
        ATSCtl: Enable-, Smallest Translation Unit: 00
    Capabilities: [640 v1] Page Request Interface (PRI)
        PRICtl: Enable- Reset-
        PRISta: RF- UPRGI- Stopped+
        Page Request Capacity: 0000071f, Page Request Allocation: 00000000
    Capabilities: [684 v1] L1 PM Substates
        L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1- L1_PM_Substates+
              PortCommonModeRestoreTime=106us PortTPowerOnTime=10us
        L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
               T_CommonMode=0us
        L1SubCtl2: T_PwrOn=10us
    Capabilities: [a18 v1] Physical Layer 16.0 GT/s <?>
    Capabilities: [e40 v1] Lane Margining at the Receiver <?>
    Capabilities: [e70 v1] Data Link Feature <?>
    Capabilities: [e80 v1] Extended Capability ID 0x2a
    Capabilities: [ea8 v1] Extended Capability ID 0x31
    Capabilities: [ec0 v1] Extended Capability ID 0x32
    Capabilities: [f00 v1] Extended Capability ID 0x33
    Capabilities: [f2c v1] Extended Capability ID 0x34
    Capabilities: [fc8 v1] Extended Capability ID 0x2e