Visibility into the FPGA.

Posts Tagged burst

Record FPGA data during 1 hour – really

Record data from FPGA over long total times - deep capture

Record FPGA data during 1 hour – really.

As ASIC, SoC and FPGA engineers, we are used to watching the operation of our designs based on single limited snapshots. RTL simulations, for instance, provide bit-level details during execution times that span over a few (milli)seconds at best. Consequently, it may not be possible to see events that happen over long times as a single coherent capture.

In this blog post, I have wanted to show what can be done in a real case with EXOSTIV. The design that runs from a FPGA board is a full system on-chip that features a Gbit Ethernet connection. The board is connected to our company network – and I have set up EXOSTIV to trigger and record the Ethernet traffic during 1 full hour. Yes, we used EXOSTIV as an Ethernet sniffer, that works from inside the FPGA – providing a ‘decoded view’ of the traffic after it has entered the FPGA Gbit Ethernet IP. Check the video below. We have accelerated it but the stop watch shows the real elapsed time.

Like it is shown on the video, we have set up EXOSTIV to record a chunk of data (256 samples) every time ‘something’ is seen on the Ethernet interface: there is an event trigger when a wide variety of events (note the ‘OR’ equation trigger definition) marking the detection of traffic happen on the Ethernet connection. Each sample is 1,248 bits wide.. This gives a quite good insight of what is ‘after’ the Gbit Ethernet IP in the FPGA. Each burst records 256 x 1,248 bits = 319,488 bits of data – and there are 30,000 such bursts collected over a total time of about 1 hour.
In total, EXOSTIV has collected a little less than 1.2 Gigabyte during this run – which is just about 15% of the total memory provided by an EXOSTIV probe.

Practically, it means that we could collect roughly 6.6 times as much data… (> 6 hours) with this single capture unit.

You can see that the bursts are recorded rather randomly, as the occurrence of a trigger depends on the actual load of our company network. (Click on pictures below to enlarge and zoom on the data)

Exostiv Dashboard after capturing 1 hour of the Ethernet traffic

Exostiv Dashboard after capturing 1 hour of the Ethernet traffic - full capture
Exostiv Dashboard after capturing 1 hour of the Ethernet traffic - detail on the deep capture

As always, thank you for reading (and watching).
– Frederic

Deep Trace & Bandwidth

Exostiv provides deep trace AND bandwidth for maximal FPGA visibility

Deep Trace & Bandwidth

Exostiv provides the following maximum capabilities for capturing data from inside FPGA running at speed:

Capabilities.

  • 50 Gigabit per second bandwidth for collecting FPGA traces.
  • 8 Gigabyte of memory for trace storage.
  • 32,768 nodes probing simultaneously.
  • 524,288 nodes reach.
Actually, we have built EXOSTIV to provide VISIBILITY to FPGA designers performing debugging with real hardware. If you do not know why it is important, watch the following 7 minutes video. It sketches out the fundamentals of EXOSTIV.

We built EXOSTIV to provide visibility to the FPGA designer.

 

 

Interrupted captures are very useful too !

I was recently demonstrating Exostiv at a customer’s site and I received the following comment:
“Even with 50 Gbps bandwidth, this tool is hardly usable because you won’t see many nodes at a usual FPGA internal sampling frequency…”
This person was implying that – for example – probing more than 250 FPGA nodes at 200 MHz already exceeds this total bandwidth. So, Exostiv cannot be used to its fullest, right?

Wrong.

This reasoning is right if you think that only continuous captures are valuable for getting insight from FPGA.
The following short video explains why it is important. It features a case where the capture – from start to end – spans over 11 seconds ! . Depending on the trigger and data qualification (or data filtering options) – and by using the full provided trace data buffer (8GB) such an approach can let you observe specific moments of the FPGA in operation over hours !.
 

 

With the proper capture settings, EXOSTIV lets you observe FPGA over hours.

So, the features listed below are equally important for an efficient capture work.

Features.

  • 16 capture units that can be enabled/disabled dynamically
  • 16 multiplexed data groups per capture unit
  • 8k samples local buffer in each capture unit.
  • 1 trigger unit per capture unit. Defines start of capture.
  • Bit or bus condition. =, /=, <, >, range, out of range conditions
  • Repeating/interrupted capture based on trigger condition
  • Data qualification condition on input data. Capture only when the condition is true.
  • Interactive trigger or data qualification definition: no recompile needed
  • Sequential / state machine trigger in 2017 roadmap.

As always, thank you for reading (and for watching)
– Frederic

Debugging FPGAs at full speed

High bandwidth

Debugging FPGAs at full speed

In my previous post, I explained why increasing the available ‘window of visibility’ is a gigantic advantage when tracking system-level issues on modern complex FPGAs. EXOSTIV’s structure does not require the FPGA internal memory to grow with the depth of the capture.

Yet, mobilizing hardware resources (internal / external hardware and software, memory, logic, communication ports, …) is always the result of a compromise. Debugging FPGA at full speed requires adopting the right strategies to remove bottlenecks.

Debugging FPGA at full speed requires adopting the right strategies to remove bottlenecks.

Memory severely limits JTAG instrumentation

In traditional JTAG-based embedded instrumentation, the main bottleneck is FPGA memory. There is some ability to scale up the FPGA memory but this depends on what’s available in the FPGA after the design is implemented. Collecting a larger number of signals and deeper traces quickly exhausts the memory. Beyond a few (hundreds of?) kilobytes at best, you’re done. This severely limits the ability to debug further.

A common strategy for scaling up debug without increasing the memory consists in multiplexing data groups (see figure below), based on the principle that debugging FPGA sometimes requires to ‘overinstrument’ the design, to save on implementation iterations. ‘Multiplexing’ means segmenting the design into groups of signals. This technique helps reserve the scarce memory resources for a reduced set of signals, thereby enabling longer (‘deeper’) captures.

A displacement towards bandwidth

Unlike JTAG-based systems – where the captured data is really ‘trapped’ inside the FPGA – EXOSTIV sends the captured data to a large external memory (8 GB currently). This memory is hardly the bottleneck and it can be scaled up rather easily.

For EXOSTIV, the ‘new’ bottleneck is the available bandwidth (please note that even with this bottleneck, EXOSTIV goes much beyond the capabilities of JTAG-based embedded instrumentation).
Advanced versions of EXOSTIV provide up to 50 Gbps of total bandwidth with 8 GB of memory. However, these resources are in no way ‘unlimited’ in regard of the ‘exploding complexity’ of modern FPGA. 50 Gbps allows the continuous streaming of 1,000 bits sampled at… just 50 MHz.

Pushing the boundaries

EXOSTIV includes the following features to preserve bandwidth:

  • Data group multiplexing;
  • Data qualification;
  • Burst mode of operation;

All of the above strategies act differently with one purpose: reduce the bandwidth requirements.

Data Group Multiplexing

We have evoked this strategy previously, as a way to ‘deepen’ captures for JTAG-based systems, where the storage memory is very small.
For EXOSTIV, multiplexing the observed nodes as data groups has the function of reducing the maximal bandwidth required to flow the data out of the FPGA. While it constraints the designer to watch the data groups separately, it helps reduce the bandwidth requirements to this available on the gigabit links. If this bandwidth is kept below or equal to the maximum available, ‘continuous’ capture is possible, and data can be flown outside of the FPGA without requiring any local storage. The process can stop when the EXOSTIV Probe’s memory is full.

Data Qualification

Data qualification consists in defining ‘filters’ or ‘qualification equations’ built on the observed data. A logic condition is defined, which enables or disables the capture of data. This equation defines when ‘data is (un)interesting’. For instance, you may want to observe a data bus only when there is traffic on it and filter the IDLE times.
Coupled with some buffering in EXOSTIV IP Capture Units (click here for more details on EXOSTIV IP’s structure), this strategy effectively contributes to lowering the average bandwidth requirements.

Burst mode of operation.

Capturing repetitive chunks of data defined with a trigger condition is another solution. Like data qualification, bursting data contributes to lowering the average bandwidth requirements on the gigabit transceiver links. Using bursts must not be seen as a ‘fallback strategy’ for cases when streaming data continuously is not possible. As opposed to JTAG-based solutions, where only one single burst can be observed, EXOSTIV allows bursting data out until the Probe’s memory is full. Events occurring between bursts cannot be watched, but a large set of events that happen long in time after power up are now accessible (Remember: with JTAG-based systems, all you can see is… just one burst!).

Take away : make the most of your resources

EXOSTIV provides unprecedented memory and bandwidth resources that enable debugging modern FPGAs at full speed.
These already huge resources can be scaled with modern FPGA complexities. In addition, using the right EXOSTIV features and strategies help remove a ‘new debugging bottleneck’: bandwidth. Combining burst mode operation, data qualification and data grouping is a winning strategy to make the most of EXOSTIV’s debugging resources.

Thank you for reading.
– Frederic

You can capture tons of data. Now what?

EXOSTIV can capture very large data sets

You can capture tons of data. Now what?

Offering huge new capabilities is not always seen positively. Sometimes, engineers come to us and ask:
‘Now that I am able to collect Gigabytes of trace data from FPGA running at speed… how do I analyze that much data?’.

Does EXOSTIV just displace the problem?

No, it does not.

In this post, I’ll show:
1) why it is important to enlarge the observation window. – and:
2) how to make the most of this unprecedented capability, based on EXOSTIV Dashboard’s capture modes, triggers and data qualification capabilities.

Random bugs happen

The figure below shows a typical debug scenario. A bug occurs randomly ‘some time’ after system power up. During an initial time interval, the effects of the bug are not noticeable – after which, the system’s behaviour is sufficiently degraded to obviously see the bug’s effect. Typically, the engineer will need to detect a ‘start of investigation’ condition from which he’ll have to ‘roll back in time’ – ultimately to find the root cause of the bug. In the case of an FPGA the engineer observes the FPGA’s internal logic.

Random bugs typically occur as a result of a simulation model issue. The system’s environment could not be simulated with complete accuracy and/or some part of the system behaves differently than initially thought int he ‘real world’. In addition, long operating times are not really the ‘simulation’s best friend’.

Size matters.

Suppose that we capture some ‘observation window’ of the system. If we are lucky enough to observe a bug right when it occurs, we can get an understanding of what happens. Similarly, we are able to recognize the effects of the bug, as shown by the orange ‘X’ on the picture (called ‘start of investigation). Conversely, we have no idea about how to trigger on such conditions (if we had the ability to trigger on unknown and random bugs, they would have been corrected already).

With a reduced observation window, the process of debugging requires some luck. We can miss everything, recognize an effect without having sufficient data to find the start of investigation. We can be lucky enough to find the start of investigation but no history to roll back to the root cause… – Or: we can hope to ‘shoot right on the bug’.

You do not want to leave the debugging process to chance?

Enlarge the observation window!

VS.


With a properly sized observation window, you can be certain of finding a random bug.

Not ‘brute force’ only

EXOSTIV provides a total observation window of 8 Gigabytes.
However, EXOSTIV is not a ‘brute force’ solution only. Many of its features are equally important to implement ‘smarter’ captures in the lab.

Trigger, Data Qualification, Streaming and Burst help you define better capture windows.

1. Trigger

The total capture window may be gigantic, a triggering capability remains essential. During EXOSTIV IP setup, the nodes that need to be observed are optionally marked as ‘source for trigger’. Currently, EXOSTIV is able to implement a single or repeating trigger based on AND and OR equations defined on these signals. The AND and OR equations can also be combined. (In a future release, EXOSTIV will provide sequential triggers as well.)

With the growing complexity of systems, defining a very accurate trigger condition has sometimes become a way to overcome a too limited capture window.

This time is over. You are now able to use trigger to collect a better capture window. Maybe you can relax a little bit about defining complex triggering, because you can count on a very large capture window to collect the information that you need.

2. Data Qualification

‘Data qualification’ enables you to define a logic condition on the observed signals. Data is captured and recorded by EXOSTIV only when this condition is true.

For instance, this can be used to filter out the times when nothing happens on a bus. This is a very powerful feature that helps you preserve the bandwidth that you use to extract data.

3. Streaming or Burst?

EXOSTIV provides 2 modes of capture:

  • Streaming mode: data is captured continuously. If the bandwidth required to capture data exceeds its EXOSTIV Probe capabilities, it stops and resumes capture once it is able to do so. Triggers and capture length can be used to define a repeating start condition.
  • Burst mode: data is captured by ‘chunks’ equal to the ‘burst size’. The maximum burst size is defined by the size of the Capture Unit’s FIFO (click here for more details about EXOSTIV IP). So, ‘burst mode’ is always an ‘interrupted’ type of capture.


Takeaways

Using EXOSTIV reduces to 2 types of capture: continuous or interrupted.
‘Continuous capture’ is easy to understand: EXOSTIV starts capturing data and has sufficient bandwidth on its transceivers to sustain data capture until its large probe memory is full.

This is an unprecedented capability for a FPGA debug tool.

‘Interrupted capture’ – whether based on trigger, data qualification, burst mode or a combination of them – help ‘better capture windows’ that focus on data of interest. Interrupted captures also help go beyond the maximum bandwidth of the transceivers and capture data even farther in time than what a continuous capture would allow.

This considerably extends your reach in time, after starting up the FPGA.

Thank you for reading.
– Frederic

STAY IN TOUCH

Sign in to our Newsletter