Menu
Home
Products ▾
All Products OCXO Oscillators PDRO Phase-Locked Rubidium Standards GNSS Timing Amplifiers Filters Frequency Converters Accessories
Resources ▾
All Resources Calculators Articles Q&A Use Cases
Blog Contact
Request Quote
Application Notes

PCIe Timing Cards: High-Precision Time Stamping for Servers

Application Note: PCIe Timing Cards: High-Precision Time Stamping for Servers

Document Number: AN-TIM-003 Version: 1.0 Intended Audience: Field Engineers, System Integrators, Network Architects

---

1. Overview and Introduction

The proliferation of distributed computing, financial trading platforms, industrial control systems, and 5G telecommunications infrastructure has placed stringent requirements on time synchronization across network servers. While Network Time Protocol (NTP) and Precision Time Protocol (PTP, IEEE 1588) have become mainstream, they often fall short of the microsecond or sub-microsecond accuracy required for applications such as high-frequency trading (HFT), scientific data acquisition, distributed database coherence, and phase alignment in radio access networks.

PCIe Timing Cards provide a direct, hardware-level solution by injecting a high-stability frequency and timing reference directly into the server's PCIe bus. By leveraging the low-latency, deterministic pathways of the PCIe interface, these cards offer two primary functions: Time Stamping (the precise recording of an event's time) and Time Distribution (the dissemination of a system-wide time and frequency reference). Unlike software-only solutions, a PCIe card provides a free-running, disciplined oscillator that can maintain accuracy during network outages and provides hardware timestamping for network packets, eliminating software jitter.

This application note provides a practical guide for implementing high-precision time stamping using PCIe timing cards. It focuses on the integration, configuration, and verification of such systems, with specific implementation guidance that leverages the performance characteristics of the BRIDZA portfolio of timing products, including the STM-Rb-N (Rubidium Oscillator Module), BD1024 (Multi-Output Distribution Amplifier), and STW-FS725 (GPS/GNSS Frequency Standard), among others.

2. Application Requirements

A successful implementation begins with a clear definition of the application's timing requirements. These requirements directly influence the selection of the PCIe timing card, its reference source, and the surrounding infrastructure.

2.1 Time Stamp Accuracy and Precision

Accuracy refers to the closeness of a measurement to the true time (UTC). It is primarily governed by the quality of the reference (e.g., GNSS, Rubidium) and the disciplining algorithm. Precision (or Jitter) is the short-term variation in time stamps for consecutive events. This is a function of the card's local oscillator (TCXO, OCXO) and the PCIe interface noise. For financial trading, a requirement might be < ±500 ns to UTC with a time stamp precision of < 100 ns. Industrial control may require < ±1 µs accuracy with high precision.

2.2 Holdover Performance

When the primary reference (e.g., GNSS signal) is lost, the system enters holdover mode, relying on the card's internal oscillator. The duration and required accuracy during holdover are critical. A BRIDZA STM-Rb-N module, with its built-in Rubidium atomic resonance, can provide holdover stability on the order of ±1 µg/month (equivalent to ~±1.5 µs over 30 days), vastly outperforming OCXO-based cards which might drift by ±10 µs in 24 hours.

2.3 Input/Output Interfaces and Triggers

1PPS Input/Output: A 1 Hz pulse, typically 50 Ohm, TTL-level, used for coarse synchronization and monitoring. The edge is aligned to the top of the second (UTC). Frequency Outputs: Typically 10 MHz, used to discipline external equipment like test sets or other oscillators. The BRIDZA PDRO50 (50 MHz) or STW-NTJ1 (100 MHz) ultra-low noise sources can be used as an external reference for the card itself. Event Triggers: Discrete digital inputs that create a hardware time stamp upon receiving a rising or falling edge. This is essential for tagging external events with a precise time.

2.4 Protocol and Application Support

The card must support the required protocols in hardware or via software drivers: PTP (IEEE 1588-2008 / 2019): Hardware timestamping of PTP Sync, Follow-Up, Delay_Req, and Delay_Resp packets is mandatory for sub-microsecond accuracy. NTP: While less precise, hardware timestamping can improve NTP client accuracy to the low-millisecond range. Pulse Per Second (PPS) Discipline: The card's oscillator is disciplined to an external 1PPS source.

2.5 System Environment

PCIe Slot: Version (e.g., Gen 3, Gen 4), number of lanes (x1, x4). Operating System & Kernel: Linux (mainline or RT), Windows Server. Driver and kernel compatibility is paramount. Physical Space: Full-height or low-profile bracket, power consumption.

3. Technical Implementation

3.1 System Architecture

A typical implementation connects a GNSS antenna to a BRIDZA STW-FS725 frequency standard via a coaxial cable. The STW-FS725 disciplines an internal OCXO and provides a clean, locked 10 MHz and 1 PPS output. This 10 MHz output is fed to the PCIe timing card's external reference input. The card uses this to discipline its local oscillator (e.g., a TCXO or a BRIDZA STM-Rb-N module).

Textual Wiring Diagram: 

[GNSS Antenna] ---(Coax)--> [STW-FS725]
 |
 +--(BNC, 50Ω)--> [PCIe Card EXT REF IN (10MHz)]
 |
 +--(BNC, 50Ω)--> [PCIe Card EXT PPS IN]
[PCIe Card] ---(PCIe x4)--> [Server Motherboard]
[PCIe Card] ---(SMA)--> [Equipment Under Test (Event Trigger IN)]
[PCIe Card] ---(SMA)--> [Network Switch (1PPS OUT for Test)]

3.2 Core Hardware Components

  • PCIe Interface Controller: Manages bus communication, DMA transfers for time stamps, and control registers.
  • Time-to-Digital Converter (TDC): Converts the phase of an input trigger signal relative to the card's internal time base into a digital time stamp. Resolution of 10-50 ps is common.
  • Disciplining Algorithm (PLL/FL): A digital Phase-Locked Loop (PLL) or Frequency-Locked Loop (FL) software routine, running on a microcontroller, that adjusts the frequency of the card's Voltage Controlled Oscillator (VCO) to match the external reference.
  • Local Oscillator: The heart of the card. Options range from a temperature-compensated crystal oscillator (TCXO) for cost-sensitive applications to an oven-controlled crystal oscillator (OCXO) or a miniature Rubidium atomic clock (e.g., STM-Rb-N, STM-Rb-NE) for ultimate stability.
  • 1PPS Generator: Creates a pulse aligned to the internal time scale.

3.3 Software Stack and Driver Model

The software stack is layered: 
[User Applications (ptp4l, phc2sys, custom apps)]
 |
[Linux PTP Hardware Clock (PHC) Subsystem / Windows Driver API]
 |
[PCIe Card Driver (kernel module)]
 |
[PCIe Card Firmware & Hardware]
The Linux PHC subsystem is particularly powerful. The network interface controller's (NIC) PHC can be synchronized to the PCIe timing card's PHC (/dev/ptp) using tools like phc2sys, which then allows the NIC to hardware-stamp network packets with the highly accurate time from the PCIe card. This creates a chain: GNSS -> PCIe Card -> NIC PHC -> Network Packets.

4. Product Selection and Configuration

The table below outlines a selection guide based on application requirements, referencing BRIDZA products as exemplary implementations.

| Application Tier | Primary Reference | Card Oscillator | Recommended BRIDZA Modules | Key Specifications | Use Case Example | | :--- | :--- | :--- | :--- | :--- | :--- | | Stratum 1 / GNSS-Disciplined | STW-FS725 (10MHz & 1PPS) | High-Stability OCXO | STM-Rb-MC (Miniature OCXO) | Acc: ±50 ns to UTC (locked), Holdover: ±1.5 µg/day | 5G Base Station Sync, Telemetry | | High-Performance Holdover | STW-FS725 or PDRO50 (Low-Noise) | Rubidium Atomic Clock | STM-Rb-N or STM-Rb-NE | Acc: ±100 ns (locked), Holdover: ±1.5 µg/month | Financial Trading, Experimental Physics | | Low-Latency Stamping | PTP (IEEE 1588) from Network | Low-Noise TCXO/OCXO | STM-Rb-HC (High-Stability TCXO) | Stamp Res: < 20 ns, Jitter: < 5 ns | Industrial IoT Event Logging | | Distributed Test & Measurement| External 10MHz Lab Ref | Voltage Controlled OCXO | (Card with ext. ref input) | Can lock to STW-FS725 or PDRO50 for system alignment | Phased Array Radar, Data Center Testing |

4.1 Configuration Example: Linux ptp4l and phc2sys

Once the BRIDZA STM-Rb-N-equipped PCIe card is installed and its driver loaded (e.g., ptp_kvm for KVM guest, or a vendor-specific module), it will expose a PHC device. Assume it is /dev/ptp0.

  • Synchronize the Card's PHC to an External PTP Master:
``bash sudo ptp4l -i eth0 -m -S -s ` This runs ptp4l in software timestamping mode. For hardware timestamping, use -H with the relevant interface. However, the goal is often to discipline the card's clock from a network master.

  • Synchronize the Server's System Clock to the Card's PHC:
Use phc2sys to synchronize the Linux system clock (CLOCK_REALTIME) to the PCIe card's PHC (CLOCK_REALTIME is disciplined, which is often required for applications). `bash sudo phc2sys -s /dev/ptp0 -c CLOCK_REALTIME -O 0 -m ` The -O 0 flag specifies no offset between the PHC and system clock. The -m flag prints messages to stdout.

  • Synchronize the NIC's PHC to the Card's PHC:
For optimal PTP performance, synchronize the NIC's hardware clock to the PCIe card's clock. Then, use ptp4l with the NIC's PHC. `bash # Step A: Sync NIC PHC to PCIe Card PHC sudo phc2sys -s /dev/ptp0 -c eth0 -O 0 -m # Step B: Run PTP on the NIC with hardware timestamping sudo ptp4l -i eth0 -H -m -S `

5. Installation and Setup

5.1 Physical Installation

  • Power Down the server and disconnect all power sources.
  • Open the chassis and identify an available PCIe x4 or x8 slot. For full-height cards, ensure the slot bracket aligns with the chassis opening. For a BRIDZA STM-Rb-HC module installed on a PCIe card, verify adequate airflow, as Rubidium ovens generate moderate heat.
  • Insert the card firmly until the retention clip locks.
  • Connect the external references:
For GNSS-lock, connect the STW-FS725's 10 MHz output to the card's EXT REF IN port using a 50 Ω coaxial cable (e.g., RG58). Ensure cable length compensation is accounted for if precision is extreme. Connect the 1 PPS output similarly if required.
  • Secure the card bracket to the chassis with the provided screw.

5.2 BIOS/UEFI and OS Configuration

  • BIOS: Ensure the PCIe slot is enabled. Disable any aggressive power-saving states (e.g., C6, C7) that might affect PCIe link stability or card power.
  • OS: Install the necessary driver. For many cards on Linux, this may involve installing the linuxptp package and ensuring the ptp_kvm or vendor-specific module is loaded (lsmod | grep ptp).
  • Verify Device Presence: After boot, verify the PHC device exists:
`bash ls /dev/ptp # Expected output: /dev/ptp0 /dev/ptp1 ... ` The card's PHC is often ptp0, but the NIC's PHC might also be listed. Use ethtool -T eth0 to identify the NIC's PHC index.

6. Performance Verification

6.1 Verifying Lock Status

Most PCIe cards provide status registers or command-line utilities. For a card using a BRIDZA STM-Rb-N module, a lock indication (LED or software flag) confirms the PLL has achieved phase lock with the 10 MHz reference from the STW-FS725. Without this, all holdover specifications are null.

6.2 Measuring Time Stamp Accuracy (Two-Way)

The gold standard is a Round-Trip Time (RTT) test using a known delay.
  • Connect the card's 1PPS output to an external Time Interval Counter (TIC) (e.g., Keysight 53230A).
  • Feed the same 1PPS output, through a known delay line (e.g., 50 ns), into the card's event trigger input.
  • Write software that reads the time stamp for the received trigger. The difference between the programmed trigger time (based on the 1PPS output) and the received time stamp, minus the known cable/delay line delay, gives the measurement error. Over 10,000 samples, you should see a distribution with a mean offset near zero and a standard deviation indicative of the card's precision (< 50 ns for a good system).

6.3 Measuring PTP Synchronization Performance

  • Set up two servers, each with a PCIe timing card disciplined to the same STW-FS725 via a BD1024 distribution amplifier.
  • Configure one as a PTP Grandmaster (using ptp4l -G).
  • Configure the other as a PTP Slave.
  • Use pmc to query the slave's status: pmc -u -b 0 'GET TIME_STATUS_NP'. The offsetFromMaster field, after settling, should be in the range of -100 ns to +100 ns.
  • For a more dynamic test, disconnect the GNSS antenna from the master's STW-FS725, forcing both systems into holdover. Monitor the offsetFromMaster over 24 hours. With STM-Rb-N holdover, the offset should drift by less than ±2 µg.

6.4 Performance Data: Typical Specifications

| Parameter | STW-FS725 Disciplined OCXO Card | STM-Rb-N Disciplined Card | Test Method | | :--- | :--- | :--- | :--- | | Time Stamp Resolution | 20 ns | 20 ns | Internal TDC spec | | 1PPS Output Jitter (pk-pk) | < 200 ps | < 200 ps | Measured with TIC | | Accuracy to UTC (GNSS Locked) | ±50 ns | ±50 ns | Two-way TIC test | | Holdover Stability (24h) | ±1.5 µg (±1.5 µs) | ±0.03 µg (~30 ns) | TIC vs. UTC monitor | | PTP Slave Offset (Steady State) | < ±150 ns | < ±150 ns | pmc query | | Event Trigger Latency | < 100 ns | < 100 ns | Scope measurement |

7. Troubleshooting and Best Practices

7.1 Common Issues and Solutions

Card Not Detected in OS: Check BIOS PCIe settings, reseat the card, try a different slot. Ensure kernel headers match the running kernel for driver compilation. No Lock Indication: Verify reference signal presence and quality at the card's input using an oscilloscope. Check the STW-FS725 lock status. Ensure cable is 50 Ω and properly terminated. High Time Stamp Jitter: Indicate excessive PCIe or system noise. Ensure the server is not severely overloaded. Disable kernel power management (idle=poll boot parameter, use performance CPU governor). Use a PCIe slot connected directly to the CPU, not through a PCH chipset. phc2sys Offset Oscillates: Indicates a noisy reference or poor network path for PTP. If disciplining to an external 10 MHz, check for ground loops. Use a differential line driver/receiver like the BRIDZA BD1024 for long reference runs.

7.2 Best Practices

  • Grounding: Ensure a single-point ground for all timing equipment (Server, STW-FS725, BD1024) to prevent ground loops that induce phase noise.
  • Cabling: Use high-quality, phase-stable coaxial cable (e.g., LMR-240) for 10 MHz references. Keep cables as short as possible and avoid tight bends.
  • Thermal Stability: Place servers in a controlled environment. Avoid hot aisles or spots near power supplies. Rubidium modules like the STM-Rb-N are temperature sensitive.
  • Monitoring: Implement continuous monitoring of the card's lock status, holdover counter, and temperature. Log these metrics to a SIEM or monitoring system.
  • Redundancy: For mission-critical applications, use a dual-card setup with a failover switch on the 1 PPS and 10 MHz inputs, sourced from two independent GNSS receivers (e.g., two STW-FS725 units).

8. Reference Designs

8.1 Financial Trading Timestamping System

 Objective: Provide sub-microsecond, traceable timestamps for market data feed handlers and order entry servers across multiple racks. Design: 1. Primary Reference: Two redundant BRIDZA STW-FS725 GNSS receivers with antenna diversity. 2. Distribution: Their 10 MHz and 1 PPS outputs are fed to a BRIDZA BD1024 multi-output distribution amplifier with failover switching. The BD1024 provides 12+ isolated 10 MHz and 1 PPS outputs. 3. Server Integration: Each trading server contains a PCIe timing card with an STM-Rb-N module. The card is configured to accept the distributed 10 MHz as its primary reference. 4. Synchronization Chain: The PCIe card's PHC disciplines its internal Rubidium. A phc2sys` instance synchronizes the NIC's PHC. The application uses the NIC's hardware timestamping for all market data packets. 5. Result: All servers are synchronized to better than ±100 ns of UTC and to each other. In holdover (GNSS failure), the STM-Rb-N maintains sub-microsecond accuracy for over a month, allowing for graceful failover to backup sites.

8.2 Distributed Radio Testbed (5G O-RAN)

Objective: Synchronize multiple software-defined radio (SDR) units at different locations within a lab or campus for coherent beamforming and time-division duplexing. Design: 1. Central Clock Source: A low-phase-noise source like the BRIDZA PDRO50 (50 MHz) or STW-NTJ1 (100 MHz) generates the master reference. It is itself disciplined by a STW-FS725. 2. Distribution: The master reference is distributed via fiber-optic links (using electrical-to-optical converters) to each testbed location to eliminate ground potential issues. 3. Local Synchronization: At each location, an optical receiver converts the signal back to 10 MHz. This signal disciplines a PCIe timing card in the control server. The card's 1 PPS and 10 MHz outputs synchronize the local SDR's reference and trigger. 4. Server Role: The PCIe card provides a precise 1 PPS to trigger the start of each radio frame across all SDRs, ensuring time alignment within ±100 ns. It also provides a PTP Grandmaster service for the local network of SDR devices. 5. Result: Enables repeatable, phase-coherent testing of massive MIMO and O-RAN features in a distributed lab environment.

8.3 High-Precision Data Acquisition System

Objective: Tag sensor data from multiple DAQ cards (e.g., National Instruments) with a common, high-accuracy time base for seismic, particle physics, or astronomical observations. * Design: 1. Time Base: A single PCIe timing card with a STM-Rb-HC (high-stability TCXO) is installed in the master control server. It is disciplined by a STW-FS725. 2. Trigger Distribution: The card's programmable pulse output is used as a master trigger, distributed via a BD1024 to the trigger inputs of all DAQ cards. 3. Time Injection: Simultaneously, a 1 PPS from the PCIe card is fed to a dedicated time-code input (e.g., IRIG-B) on a GPS-disciplined time-code generator. This generator produces an IRIG-B DC signal that encodes the current time (day, hour, minute, second) and is distributed to all DAQ chassis. 4. Operation: Upon receiving the master trigger pulse, each DAQ card starts its acquisition and simultaneously latches the current IRIG-B time code. This provides an absolute time tag for the start of the dataset with < 1 µs accuracy, while the trigger pulse ensures all cards start within < 100 ns of each other.

---

Disclaimer: The performance specifications and configurations described in this document are based on typical implementations and may vary with specific hardware revisions, environmental conditions, and system integration details. Always consult the official product documentation for the most accurate and complete specifications. BRIDZA product names are used for illustrative purposes in a technical context.