Visibility into the FPGA.

Posts Tagged observability

Deep Trace & Bandwidth

Exostiv provides deep trace AND bandwidth for maximal FPGA visibility

Deep Trace & Bandwidth

Exostiv provides the following maximum capabilities for capturing data from inside FPGA running at speed:

Capabilities.

  • 50 Gigabit per second bandwidth for collecting FPGA traces.
  • 8 Gigabyte of memory for trace storage.
  • 32,768 nodes probing simultaneously.
  • 524,288 nodes reach.
Actually, we have built EXOSTIV to provide VISIBILITY to FPGA designers performing debugging with real hardware. If you do not know why it is important, watch the following 7 minutes video. It sketches out the fundamentals of EXOSTIV.

We built EXOSTIV to provide visibility to the FPGA designer.

 

 

Interrupted captures are very useful too !

I was recently demonstrating Exostiv at a customer’s site and I received the following comment:
“Even with 50 Gbps bandwidth, this tool is hardly usable because you won’t see many nodes at a usual FPGA internal sampling frequency…”
This person was implying that – for example – probing more than 250 FPGA nodes at 200 MHz already exceeds this total bandwidth. So, Exostiv cannot be used to its fullest, right?

Wrong.

This reasoning is right if you think that only continuous captures are valuable for getting insight from FPGA.
The following short video explains why it is important. It features a case where the capture – from start to end – spans over 11 seconds ! . Depending on the trigger and data qualification (or data filtering options) – and by using the full provided trace data buffer (8GB) such an approach can let you observe specific moments of the FPGA in operation over hours !.
 

 

With the proper capture settings, EXOSTIV lets you observe FPGA over hours.

So, the features listed below are equally important for an efficient capture work.

Features.

  • 16 capture units that can be enabled/disabled dynamically
  • 16 multiplexed data groups per capture unit
  • 8k samples local buffer in each capture unit.
  • 1 trigger unit per capture unit. Defines start of capture.
  • Bit or bus condition. =, /=, <, >, range, out of range conditions
  • Repeating/interrupted capture based on trigger condition
  • Data qualification condition on input data. Capture only when the condition is true.
  • Interactive trigger or data qualification definition: no recompile needed
  • Sequential / state machine trigger in 2017 roadmap.

As always, thank you for reading (and for watching)
– Frederic

You can capture tons of data. Now what?

EXOSTIV can capture very large data sets

You can capture tons of data. Now what?

Offering huge new capabilities is not always seen positively. Sometimes, engineers come to us and ask:
‘Now that I am able to collect Gigabytes of trace data from FPGA running at speed… how do I analyze that much data?’.

Does EXOSTIV just displace the problem?

No, it does not.

In this post, I’ll show:
1) why it is important to enlarge the observation window. – and:
2) how to make the most of this unprecedented capability, based on EXOSTIV Dashboard’s capture modes, triggers and data qualification capabilities.

Random bugs happen

The figure below shows a typical debug scenario. A bug occurs randomly ‘some time’ after system power up. During an initial time interval, the effects of the bug are not noticeable – after which, the system’s behaviour is sufficiently degraded to obviously see the bug’s effect. Typically, the engineer will need to detect a ‘start of investigation’ condition from which he’ll have to ‘roll back in time’ – ultimately to find the root cause of the bug. In the case of an FPGA the engineer observes the FPGA’s internal logic.

Random bugs typically occur as a result of a simulation model issue. The system’s environment could not be simulated with complete accuracy and/or some part of the system behaves differently than initially thought int he ‘real world’. In addition, long operating times are not really the ‘simulation’s best friend’.

Size matters.

Suppose that we capture some ‘observation window’ of the system. If we are lucky enough to observe a bug right when it occurs, we can get an understanding of what happens. Similarly, we are able to recognize the effects of the bug, as shown by the orange ‘X’ on the picture (called ‘start of investigation). Conversely, we have no idea about how to trigger on such conditions (if we had the ability to trigger on unknown and random bugs, they would have been corrected already).

With a reduced observation window, the process of debugging requires some luck. We can miss everything, recognize an effect without having sufficient data to find the start of investigation. We can be lucky enough to find the start of investigation but no history to roll back to the root cause… – Or: we can hope to ‘shoot right on the bug’.

You do not want to leave the debugging process to chance?

Enlarge the observation window!

VS.


With a properly sized observation window, you can be certain of finding a random bug.

Not ‘brute force’ only

EXOSTIV provides a total observation window of 8 Gigabytes.
However, EXOSTIV is not a ‘brute force’ solution only. Many of its features are equally important to implement ‘smarter’ captures in the lab.

Trigger, Data Qualification, Streaming and Burst help you define better capture windows.

1. Trigger

The total capture window may be gigantic, a triggering capability remains essential. During EXOSTIV IP setup, the nodes that need to be observed are optionally marked as ‘source for trigger’. Currently, EXOSTIV is able to implement a single or repeating trigger based on AND and OR equations defined on these signals. The AND and OR equations can also be combined. (In a future release, EXOSTIV will provide sequential triggers as well.)

With the growing complexity of systems, defining a very accurate trigger condition has sometimes become a way to overcome a too limited capture window.

This time is over. You are now able to use trigger to collect a better capture window. Maybe you can relax a little bit about defining complex triggering, because you can count on a very large capture window to collect the information that you need.

2. Data Qualification

‘Data qualification’ enables you to define a logic condition on the observed signals. Data is captured and recorded by EXOSTIV only when this condition is true.

For instance, this can be used to filter out the times when nothing happens on a bus. This is a very powerful feature that helps you preserve the bandwidth that you use to extract data.

3. Streaming or Burst?

EXOSTIV provides 2 modes of capture:

  • Streaming mode: data is captured continuously. If the bandwidth required to capture data exceeds its EXOSTIV Probe capabilities, it stops and resumes capture once it is able to do so. Triggers and capture length can be used to define a repeating start condition.
  • Burst mode: data is captured by ‘chunks’ equal to the ‘burst size’. The maximum burst size is defined by the size of the Capture Unit’s FIFO (click here for more details about EXOSTIV IP). So, ‘burst mode’ is always an ‘interrupted’ type of capture.


Takeaways

Using EXOSTIV reduces to 2 types of capture: continuous or interrupted.
‘Continuous capture’ is easy to understand: EXOSTIV starts capturing data and has sufficient bandwidth on its transceivers to sustain data capture until its large probe memory is full.

This is an unprecedented capability for a FPGA debug tool.

‘Interrupted capture’ – whether based on trigger, data qualification, burst mode or a combination of them – help ‘better capture windows’ that focus on data of interest. Interrupted captures also help go beyond the maximum bandwidth of the transceivers and capture data even farther in time than what a continuous capture would allow.

This considerably extends your reach in time, after starting up the FPGA.

Thank you for reading.
– Frederic

The FPGA problem we are trying to solve

Problem and solution

The FPGA problem we are trying to solve

Designing FPGA can be complex. Each step of the design flow brings its own challenges, problems and solutions. As engineers, this is what we do: we find solutions. We use our knowledge, we mobilize our skills and find the right tools to constantly build better solutions to the problems that we encounter.

Exostiv Labs aims at providing better tools for FPGA debug. However, ‘FPGA debug’ may be understood quite diversely. In this post, I’d like to go back to the basics and pinpoint the specific problems we are trying to solve.

‘First time right’ at board bring-up is a problem.

I am an engineer. Like any engineer, I like when things are done right technically, with the right steps taken at the right time.
However, as an engineer, I also have learned to manage budget and market constraints. We all know that between the ‘perfect design flow’ and the ‘actual design flow’, the distance is made of budget and/or market constraints. We all know that sometimes, it is chosen to skip the verification of one ‘detail’ in order to reach the announced release date. Sometimes it is expected from us to accept a small percentage of uncertainty (that is called ‘experience’ or ‘know-how’) because it is statistically cost-effective.

Ideal vs. Actual design flow

“And what if a bug appears later? Well, we’ll fix it and send an upgrade.
Isn’t it the beauty of programmable devices?”

Sometimes, being ‘first on the market’ or ‘first in budget’ beats ‘first time right’.

A typical in-lab situation

We all have been there…

The system starts up and suddenly, ‘something goes wrong’. It happens randomly, sometimes 2 minutes, sometimes 2 hours after power up.
– We have no clue about what caused the bug.
– We have no idea about why it happens.
– We do not know the time from cause to effect.

This is called ‘an emergent system with a ‘random bug’.

It is emergent because it is the result of complex interactions of individual pieces. Such system-level bugs are the most complex – and time-consuming – to solve because they involve the complex interactions of a whole system.

EXOSTIV solves the problem of finding the roots of bugs appearing in complex systems placed in their real environment.

EXOSTIV solves FPGA board bring up problems

Solving such a problem requires *a LOT of* visibility on the FPGA under test:
– It is hard to put a trigger condition on an unknown random bug. To spot it, you’d better have the largest capture window possible.
– You sometimes need to extend the capture far in time (stressing the impracticability of simulation as sole methodology).
– What happened before seeing the bug is very important to understand the ‘fatal’ sequence of events that has created the faulty condition.

Without a sufficient observability of the system under test, you’ll simply loose precious engineering hours hoping to -just- capture the bug. Then you’ll spend time again trying to capture the key events that happened before it – that ultimately would help you understand why the bug occurs.

Instrumenting a real design is the key to overcoming modelling mistake and see bugs occurring in the real environment… but only if the instrumented design provides sufficient visibility !

What can you do with 8 GB captured at FPGA speed of operation?

Please think to it. I’ll explore how to make the most of the *huge* resources offered by EXOSTIV in my next post.

Thank you for reading
– Frederic

‘My FPGA debug and verification flow should be improved…’

Improve the FPGA debug flow

‘My FPGA debug and verification flow should be improved…’

In my last post, I revealed some of the results of our recent survey on FPGA. These results depicted a ‘flow-conscious’ FPGA engineer, using a reduced mix of methodologies in the flow and very prone to going to the lab for debugging.

In the same survey, we tried to evaluate the level of satisfaction of the FPGA engineer for his/her debug and verification flow. We asked the respondents to select among several propositions the one that the most closely matched their thinking. See the picture below (click on picture to zoom).

Recognition of the need to improve the FPGA debug flow

More than 70% of the respondents recognize the need to improve the FPGA debug and verification flow.

The chart above represents the answers of the respondents active with an FPGA design, and actually using FPGA as a target technology. 72% of them recognize the need to improve the FPGA debug flow and nearly 40% of them are actively looking for a solution for it.

Are these survey results representative of the whole industry? Well, you tell me. Contact me to share your personal experience – and I’ll update this post.

– Oh, and by the way, as I write this, we are about to release our EXOSTIV solution. It improves the visibility on FPGA running at speed by a factor of up to 200.000 ! See below a preview of what we’re about to release. More information will be available soon.

EXOSTIV Dashboard software screenshot preview

Thank you for reading.
– Frederic

Why observability matters

Observability for typical case of FPGA-based video processing systems

Why Observability matters.

At Exostiv Labs, we think that ‘Observability’ – or ‘Visibility’ – that is ‘the ability to observe (and understand) a system from its I/Os’ – is relevant – and even key to FPGA debug.

I’d like to show it with a real example taken from the field.

A typical Video processing debugging problem

We had an issue with a FPGA-based video processing system. At some point, the ‘data header’ of a frame of a video stream running at 24 frame per second (24 fps) went wrong. The problem was rather unpredictable and occurred randomly. And -by the way- the system was not our design: when we put our hands on it, improving the design methodology to avoid bugs was not an option anymore.

Approach nr 1 : a standard embedded logic analyzer storing the captured debug information in some FPGA memory

Our engineer set up a traditional embedded logic analyzer tool to capture 1.024 bits per frame, taken from the frame header (as the content of the pictures was not very relevant to him). 1kbit per frame was deemed a rather good observability – the engineer hoped to be able to capture and understand the history of events that were creating the bug. All the captured information was stored in the FPGA memory; he was lucky enough to be able to reserve up to 32 kbit of the FPGA memory for debugging purposes.

Traditional embedded logic analyzer approach

Using this approach, our engineer has been able to observe the information related to:

32 kbit / 1.024 bit = 32 frames

At 24 fps, this is the equivalent of 1,33 second of the movie

Approach nr 2 : Sending the captured debug information to an external memory

Now, let’s say that we have an external memory as big as 8 GB and sufficient bandwidth to send the debug information ‘on-the-fly’. We’ll check the consequences on the FPGA resources later.

EXOSTIV approach
8 Gbyte equals 64 Gigabit of memory.

64 Gbit / 1.024 bit = 64.000.000 frames

64 million frames can be observed with the same ‘accuracy’ (remember that we extract only 1.024 bits from each header).
At 24 fps, this is the equivalent of : [64 M / 24] = 2,66 million second or 2,66 M / 3.600 = 740 hours of the movie !

Usually, a movie is about 2 hours long. So, we’ll be able to ‘see the full movie’ with the same ‘debugging accuracy’.

Actually, with 8 GB external memory and a 2 hours movie, we have a 740 / 2 = 370 ‘scale up factor’. This basically means that we’ll be able to extract not 1.024 bit from each frame, but 1.024 x 370 = 378.880 bits per frame.

Gigabyte range storage eliminated the ‘guess work’

We have just seen that scaling up the total capture size of the debug information has a positive impact on how much of each picture can be seen AND how much of the movie can be observed.

When a bug may occur at any moment during a full 2 hours video stream, wouldn’t you feel more secure if you were certain to have captured the bug AND the history of events that have created it?

What about the required bandwidth?

Up to this point, we have considered a very theoretical example, assuming that we would be able to stream the debug information out to the external 8 GB memory. Actually, we do have this capability – by far.

At 24 fps, extracting 378.880 bits per frame requires a total bandwidth of: 24 x 378.880 = roughly 9,1 Mbps on average.

EXOSTIV™ provides up to 8GB memory and up to 4 x 6.6 Gbps bandwidth to extract debug information from FPGA.

So…

Stop the guess work during FPGA debug,

Scale up your tools and-

Increase your observation capability.

Thank you for reading
– Frederic

STAY IN TOUCH

Sign in to our Newsletter