Visibility into the FPGA.

Posts Tagged in-lab debug

Record FPGA data during 1 hour – really

Record data from FPGA over long total times - deep capture

Record FPGA data during 1 hour – really.

As ASIC, SoC and FPGA engineers, we are used to watching the operation of our designs based on single limited snapshots. RTL simulations, for instance, provide bit-level details during execution times that span over a few (milli)seconds at best. Consequently, it may not be possible to see events that happen over long times as a single coherent capture.

In this blog post, I have wanted to show what can be done in a real case with EXOSTIV. The design that runs from a FPGA board is a full system on-chip that features a Gbit Ethernet connection. The board is connected to our company network – and I have set up EXOSTIV to trigger and record the Ethernet traffic during 1 full hour. Yes, we used EXOSTIV as an Ethernet sniffer, that works from inside the FPGA – providing a ‘decoded view’ of the traffic after it has entered the FPGA Gbit Ethernet IP. Check the video below. We have accelerated it but the stop watch shows the real elapsed time.

Like it is shown on the video, we have set up EXOSTIV to record a chunk of data (256 samples) every time ‘something’ is seen on the Ethernet interface: there is an event trigger when a wide variety of events (note the ‘OR’ equation trigger definition) marking the detection of traffic happen on the Ethernet connection. Each sample is 1,248 bits wide.. This gives a quite good insight of what is ‘after’ the Gbit Ethernet IP in the FPGA. Each burst records 256 x 1,248 bits = 319,488 bits of data – and there are 30,000 such bursts collected over a total time of about 1 hour.
In total, EXOSTIV has collected a little less than 1.2 Gigabyte during this run – which is just about 15% of the total memory provided by an EXOSTIV probe.

Practically, it means that we could collect roughly 6.6 times as much data… (> 6 hours) with this single capture unit.

You can see that the bursts are recorded rather randomly, as the occurrence of a trigger depends on the actual load of our company network. (Click on pictures below to enlarge and zoom on the data)

Exostiv Dashboard after capturing 1 hour of the Ethernet traffic

Exostiv Dashboard after capturing 1 hour of the Ethernet traffic - full capture
Exostiv Dashboard after capturing 1 hour of the Ethernet traffic - detail on the deep capture

As always, thank you for reading (and watching).
– Frederic

Deep Trace & Bandwidth

Exostiv provides deep trace AND bandwidth for maximal FPGA visibility

Deep Trace & Bandwidth

Exostiv provides the following maximum capabilities for capturing data from inside FPGA running at speed:

Capabilities.

  • 50 Gigabit per second bandwidth for collecting FPGA traces.
  • 8 Gigabyte of memory for trace storage.
  • 32,768 nodes probing simultaneously.
  • 524,288 nodes reach.
Actually, we have built EXOSTIV to provide VISIBILITY to FPGA designers performing debugging with real hardware. If you do not know why it is important, watch the following 7 minutes video. It sketches out the fundamentals of EXOSTIV.

We built EXOSTIV to provide visibility to the FPGA designer.

 

 

Interrupted captures are very useful too !

I was recently demonstrating Exostiv at a customer’s site and I received the following comment:
“Even with 50 Gbps bandwidth, this tool is hardly usable because you won’t see many nodes at a usual FPGA internal sampling frequency…”
This person was implying that – for example – probing more than 250 FPGA nodes at 200 MHz already exceeds this total bandwidth. So, Exostiv cannot be used to its fullest, right?

Wrong.

This reasoning is right if you think that only continuous captures are valuable for getting insight from FPGA.
The following short video explains why it is important. It features a case where the capture – from start to end – spans over 11 seconds ! . Depending on the trigger and data qualification (or data filtering options) – and by using the full provided trace data buffer (8GB) such an approach can let you observe specific moments of the FPGA in operation over hours !.
 

 

With the proper capture settings, EXOSTIV lets you observe FPGA over hours.

So, the features listed below are equally important for an efficient capture work.

Features.

  • 16 capture units that can be enabled/disabled dynamically
  • 16 multiplexed data groups per capture unit
  • 8k samples local buffer in each capture unit.
  • 1 trigger unit per capture unit. Defines start of capture.
  • Bit or bus condition. =, /=, <, >, range, out of range conditions
  • Repeating/interrupted capture based on trigger condition
  • Data qualification condition on input data. Capture only when the condition is true.
  • Interactive trigger or data qualification definition: no recompile needed
  • Sequential / state machine trigger in 2017 roadmap.

As always, thank you for reading (and for watching)
– Frederic

You can capture tons of data. Now what?

EXOSTIV can capture very large data sets

You can capture tons of data. Now what?

Offering huge new capabilities is not always seen positively. Sometimes, engineers come to us and ask:
‘Now that I am able to collect Gigabytes of trace data from FPGA running at speed… how do I analyze that much data?’.

Does EXOSTIV just displace the problem?

No, it does not.

In this post, I’ll show:
1) why it is important to enlarge the observation window. – and:
2) how to make the most of this unprecedented capability, based on EXOSTIV Dashboard’s capture modes, triggers and data qualification capabilities.

Random bugs happen

The figure below shows a typical debug scenario. A bug occurs randomly ‘some time’ after system power up. During an initial time interval, the effects of the bug are not noticeable – after which, the system’s behaviour is sufficiently degraded to obviously see the bug’s effect. Typically, the engineer will need to detect a ‘start of investigation’ condition from which he’ll have to ‘roll back in time’ – ultimately to find the root cause of the bug. In the case of an FPGA the engineer observes the FPGA’s internal logic.

Random bugs typically occur as a result of a simulation model issue. The system’s environment could not be simulated with complete accuracy and/or some part of the system behaves differently than initially thought int he ‘real world’. In addition, long operating times are not really the ‘simulation’s best friend’.

Size matters.

Suppose that we capture some ‘observation window’ of the system. If we are lucky enough to observe a bug right when it occurs, we can get an understanding of what happens. Similarly, we are able to recognize the effects of the bug, as shown by the orange ‘X’ on the picture (called ‘start of investigation). Conversely, we have no idea about how to trigger on such conditions (if we had the ability to trigger on unknown and random bugs, they would have been corrected already).

With a reduced observation window, the process of debugging requires some luck. We can miss everything, recognize an effect without having sufficient data to find the start of investigation. We can be lucky enough to find the start of investigation but no history to roll back to the root cause… – Or: we can hope to ‘shoot right on the bug’.

You do not want to leave the debugging process to chance?

Enlarge the observation window!

VS.


With a properly sized observation window, you can be certain of finding a random bug.

Not ‘brute force’ only

EXOSTIV provides a total observation window of 8 Gigabytes.
However, EXOSTIV is not a ‘brute force’ solution only. Many of its features are equally important to implement ‘smarter’ captures in the lab.

Trigger, Data Qualification, Streaming and Burst help you define better capture windows.

1. Trigger

The total capture window may be gigantic, a triggering capability remains essential. During EXOSTIV IP setup, the nodes that need to be observed are optionally marked as ‘source for trigger’. Currently, EXOSTIV is able to implement a single or repeating trigger based on AND and OR equations defined on these signals. The AND and OR equations can also be combined. (In a future release, EXOSTIV will provide sequential triggers as well.)

With the growing complexity of systems, defining a very accurate trigger condition has sometimes become a way to overcome a too limited capture window.

This time is over. You are now able to use trigger to collect a better capture window. Maybe you can relax a little bit about defining complex triggering, because you can count on a very large capture window to collect the information that you need.

2. Data Qualification

‘Data qualification’ enables you to define a logic condition on the observed signals. Data is captured and recorded by EXOSTIV only when this condition is true.

For instance, this can be used to filter out the times when nothing happens on a bus. This is a very powerful feature that helps you preserve the bandwidth that you use to extract data.

3. Streaming or Burst?

EXOSTIV provides 2 modes of capture:

  • Streaming mode: data is captured continuously. If the bandwidth required to capture data exceeds its EXOSTIV Probe capabilities, it stops and resumes capture once it is able to do so. Triggers and capture length can be used to define a repeating start condition.
  • Burst mode: data is captured by ‘chunks’ equal to the ‘burst size’. The maximum burst size is defined by the size of the Capture Unit’s FIFO (click here for more details about EXOSTIV IP). So, ‘burst mode’ is always an ‘interrupted’ type of capture.


Takeaways

Using EXOSTIV reduces to 2 types of capture: continuous or interrupted.
‘Continuous capture’ is easy to understand: EXOSTIV starts capturing data and has sufficient bandwidth on its transceivers to sustain data capture until its large probe memory is full.

This is an unprecedented capability for a FPGA debug tool.

‘Interrupted capture’ – whether based on trigger, data qualification, burst mode or a combination of them – help ‘better capture windows’ that focus on data of interest. Interrupted captures also help go beyond the maximum bandwidth of the transceivers and capture data even farther in time than what a continuous capture would allow.

This considerably extends your reach in time, after starting up the FPGA.

Thank you for reading.
– Frederic

The FPGA problem we are trying to solve

Problem and solution

The FPGA problem we are trying to solve

Designing FPGA can be complex. Each step of the design flow brings its own challenges, problems and solutions. As engineers, this is what we do: we find solutions. We use our knowledge, we mobilize our skills and find the right tools to constantly build better solutions to the problems that we encounter.

Exostiv Labs aims at providing better tools for FPGA debug. However, ‘FPGA debug’ may be understood quite diversely. In this post, I’d like to go back to the basics and pinpoint the specific problems we are trying to solve.

‘First time right’ at board bring-up is a problem.

I am an engineer. Like any engineer, I like when things are done right technically, with the right steps taken at the right time.
However, as an engineer, I also have learned to manage budget and market constraints. We all know that between the ‘perfect design flow’ and the ‘actual design flow’, the distance is made of budget and/or market constraints. We all know that sometimes, it is chosen to skip the verification of one ‘detail’ in order to reach the announced release date. Sometimes it is expected from us to accept a small percentage of uncertainty (that is called ‘experience’ or ‘know-how’) because it is statistically cost-effective.

Ideal vs. Actual design flow

“And what if a bug appears later? Well, we’ll fix it and send an upgrade.
Isn’t it the beauty of programmable devices?”

Sometimes, being ‘first on the market’ or ‘first in budget’ beats ‘first time right’.

A typical in-lab situation

We all have been there…

The system starts up and suddenly, ‘something goes wrong’. It happens randomly, sometimes 2 minutes, sometimes 2 hours after power up.
– We have no clue about what caused the bug.
– We have no idea about why it happens.
– We do not know the time from cause to effect.

This is called ‘an emergent system with a ‘random bug’.

It is emergent because it is the result of complex interactions of individual pieces. Such system-level bugs are the most complex – and time-consuming – to solve because they involve the complex interactions of a whole system.

EXOSTIV solves the problem of finding the roots of bugs appearing in complex systems placed in their real environment.

EXOSTIV solves FPGA board bring up problems

Solving such a problem requires *a LOT of* visibility on the FPGA under test:
– It is hard to put a trigger condition on an unknown random bug. To spot it, you’d better have the largest capture window possible.
– You sometimes need to extend the capture far in time (stressing the impracticability of simulation as sole methodology).
– What happened before seeing the bug is very important to understand the ‘fatal’ sequence of events that has created the faulty condition.

Without a sufficient observability of the system under test, you’ll simply loose precious engineering hours hoping to -just- capture the bug. Then you’ll spend time again trying to capture the key events that happened before it – that ultimately would help you understand why the bug occurs.

Instrumenting a real design is the key to overcoming modelling mistake and see bugs occurring in the real environment… but only if the instrumented design provides sufficient visibility !

What can you do with 8 GB captured at FPGA speed of operation?

Please think to it. I’ll explore how to make the most of the *huge* resources offered by EXOSTIV in my next post.

Thank you for reading
– Frederic

EXOSTIV is there – and it is not a monster

Happy Halloween

EXOSTIV is there – and it is not a monster

As you might have noticed, EXOSTIV for Xilinx is now released. With the launch, I have been on the roads to demonstrate the product.

The good thing about meeting FPGA engineers is the flurry of questions, ideas and suggestions received as I show the product. Your feedback helps us find new ideas, find where the most acute pains are and understand what you actually do. I would like to thank you, who have already dedicated some time from your supercharged week to see the product in action. (If you are interested to see the product, please contact me to check our scheduled events with me).

What is EXOSTIV?

Here is one of the slides I use to present EXOSTIV (click here for the complete presentation in PDF):

What is EXOSTIV

EXOSTIV is not an emulator.

Why is it important?

Well, because it is sometimes expected from EXOSTIV to be everything at once. Some examples:
– Can it partition design onto multiple FPGA?
(Nope, that’s the role of a partitioning tool. We have to define how our IP can be used with such tools, though).
– Can it implement this (specific) trigger condition?
(Well, some of them, some not. But with it capture capacity, you might not need such a complex trigger).
– Will it be able to replace a protocol analyzer?
(It depends on the protocol and where it is observed…).
– …

Of course, some of your suggested additional features are already in the development pipe at Exostiv Labs… But not all of them.

EXOSTIV’s main value is in the level of visibility it provides for systems running at speed.

New features will be built around this value

Ask yourself: what can you do with 8GB of captured data flowing out of your FPGA at multi-gigabit speed? Would it add something to the flow that your current tools cannot achieve?

At Exostiv Labs, we believe that a tool that tries to be everything at once would probably be very good at nothing, not well fitted to your flow and much too expensive for the value.

EXOSTIV is not such a monster.

Thank you for reading – and Happy Halloween to all!
– Frederic

STAY IN TOUCH

Sign in to our Newsletter