How to start a trace analysis for a Novulo application to find bugs

Trace analysis

When you encounter bugs or errors in an application, you need to analyze where and why the error occurs to be able to fix the bug in the architect.
It can help to enable the “trace analysis” in your application. This allows you for example to see which process is executed when, what the actual SQL queries looked like, how long each action took and what the values of variables were.

Finding the “Steps to reproduce”

If you encounter a bug as a developer, you probably know the exact steps that you took to trigger the error message. However, if a customer reports an issue the exact steps to reproduce are often unknown. It is important to try to reproduce the issue on a development instance first before enabling a trace analysis. Sometimes, this can take longer than actually fixing the bug itself. Most customers are happy to help with finding the exact steps to reproduce, so use their expertise to quickly get to the correct steps.

Starting a trace analysis

  1. Go to https://deployment.novulo.com/
  2. Open the page of the instance where you want to start the trace.

Important note:
A trace analysis should not be enabled on production applications. When a trace is started or finished, all current users are logged out. Furthermore, the performance is degraded because with each action/click, a trace file is generated. Also, after debugging, do not forget to finish the trace. Otherwise, it can happen that after a few hours, the server of your instance won’t have any memory left.

  1. Add a new installation of type “maintenance” and click “Start workflow”

  2. Open the installation in the installer and chose “[Installer] Maintenance - Trace” in the dropdown.

  3. Click on “Start workflow”. Refresh the page after a while if nothing seems to happen.

  4. Specify the options that you want to use for your trace analysis by adding “1” to the fields.If you are unsure what to choose, set a “1” at “Process” and “SQL” and a “0” at “Pause scheduling service”. When tracing REST end points, also include “Data gathering” and “Restful webservice”.

  • Cached Expressions: this one is only relevant when you are using Novulo Cached Expressions (in memory tables) in your public website or REST end points. It displays the behavior of the cache invalidation mechanism.
  • Cached Functions: this one shows the information about the calculation of cached functions. Useful when you need to find out why a cached function doesn’t have the expected value, or to analyze performance;
  • Datahandler: this one shows all the steps taken by the data handler
  • Data gathering shows the steps taken using Data Definitions, which can be used to trace individual data definition parts in REST end points
  • Export: shows the steps taken by the export engine (if used)
  • Import: shows the steps taken by the import engine (if used)
  • Inter Server Communication: shows the steps for inter server communication. Only relevant when your application is distributed over multiple web servers with local caches, as it is used in the CMS with cached expressions
  • Process: this is a textual representation of all process steps taken in the application. It’s the text that corresponds with the processes in the Architect
  • Restful webservice: displays the steps that are taken to resolve the REST end point and process parameters; for the gathering of Data definitions, also check Data gathering
  • SQL: shows all SQL queries that are being executed
  • Workflow: shows all steps taken by the workflow (if used)
  • Nr of rows: how many rows are being displayed in trace. Default = 20. More than 20 is only relevant when debugging front-end web pages that can launch more than 20 requests in one click (e.g. images)
  • Pause scheduling service: whether to pause the scheduling service while you are tracing or not
  1. Before hitting “Submit” warn other users that they will be logged out as they might otherwise lose data. You can check if any other users are logged in by putting the following expression in the expression debugger of the application.
{
 users,
 <
  ~gebruikersnaam.tostring(),
  datetime:max({
   kendoprocesslog,
   datetime,
   boolean:and([id.isgreater(int:max({kendoprocesslog,id}).sub(100)),user.equals(parent)])
  }).format(“yyyy-MM-ddHH:mm”)
 >,
 int:exists({
  kendoprocesslog,
  1,
  boolean:and([
   id.isgreater(int:max({kendoprocesslog,id}).sub(100)),
   user.equals(parent),
   boolean:not(action.equals(“nop”))
  ])
 })
}
  1. Hit submit and wait for a while before logging back into the application. It takes a few seconds for it to restart.
  2. Open a second browser tab with the URL of your application. However, replace the “default.aspx” part at the end of the URL with “trace.axd”. So https://myserver.com/myapplication/default.aspx becomes https://myserver.com/myapplication/trace.axd. A website should open with the title “Application Trace”. If you see a similar page you have successfully enabled a trace for your application.

Tracing an error

Before using the trace analysis, it is recommended to install the “Traceformatter” browser plugin for better legibility of the trace and useful features for faster debugging.

  1. Execute your steps to reproduce, except for the very last click within the application. You will see that with every click that you make, a new line will appear at in the trace.axd website, if you refresh it.

Note: Clicks from other users also trigger entries in the list. So, as a developer, you can also ask a user/customer to perform the steps to reproduce while you keep an eye on the trace analysis overview.

  1. Right before executing the last click that will trigger the error that you want to find, go to the trace page and hit “Clear all current trace data”. This will delete all previous trace files and helps to find the more interesting one.
  2. Execute the very last click in your application and wait until the error message appears, or until the spinner disappears.
  3. Manually refresh the trace website. At least one new entry should be visible in the list. The “time of request” of the entry is equal to the time when you executed your last click in the application.
  4. Click on “View details” to open the detailed trace analysis. If there are multiple entries and if you are unsure which one is the correct one, open all of them in separate tabs (e.g.: mouse wheel click) and close the ones that do not contain information related to your click.
  5. If you have the “Traceformatter” installed, you can use the buttons to collapse processes and to quickly jump to the part of the trace where the error occurred. This can look like this, for example:
  6. Scroll through the trace to analyze which actions/processes were executed and when. Use the process ids to find the related processes in the architect. Open the integrated model of you application and search for the id by using CTRL + F within the architect. Having the integrated model open next to the trace will make it easier to follow the trace step by step.

    → 1 The actual input parameters of a process become visible. Double check whether they are filled correctly. You can do so by opening the actual records in the application, for example. Based on the example screenshot, the record call with id 559 could be opened, for example.
    → 2 The start of a process is highlighted in green and are collapsible. This enables skipping parts of the process that are less interesting. They also contain the actual name of the process and the process ID at the end
    → 3 What is called a “decision” in the architect is called “transition” in the trace. If you have SQL enabled, you can see the query that was executed to transition from one action to another.
    → 4 You can find actions based on the name that they have within the architect. You can see all inputs and outputs and how long it took to execute an action in milliseconds

Stopping a trace

  1. Go to the installer where you started the trace
  2. Hit the button “Finish tracing” to disable the “trace.axd” website. All current users of the application will be logged out.
3 Likes

Is there a way to only see rows that are performed by a certain user, for example yourself or a webservice user? It’s sometimes very hard to trace something when 5 people are clicking in the application for example.

Good question, but no, unfortunately there is no way to filter traces based on the user. The only workaround right now is to ask your colleagues to stop clicking while executing step 4.
I suggest that you add a service request for your idea for the platform team.

Okay I will add service request for this. Asking other people is not always really a good option, as sometimes people are working remotely. Nevertheless, thank you for the reply.

1 Like

How to trace a ‘Object reference not set to an instance of an object’? I’ve tried different switches, but never saw this in the trace.

How to remove the translations switch? Is generates double trace lines (i think)
image

Hey Mattheüs!
To change any switch, click on the button “Redefine switches” in the installer:

Then add a value at the switch that you want to redefine.
So, “0” for “off” and “1” for “on”. If you leave the text field of a switch empty, nothing will change, and it will either stay off or on.

You can also click “Finish tracing” and start a completely new trace if the steps above do not work as expected.

The error “Object reference not set to an instance of an object” should become visible in a normal trace with the “process” switch enabled.
Since you sent a screenshot of a dialogue, I am sure that you can reproduce the error with a click in the application, so it is not a background task or something else.

Double-check the following steps for your next trace:

  1. Ask any colleagues to stop clicking in the application that you are tracing.
  2. Execute your steps to reproduce, except for the very last click within the application.
  3. Go to the trace page and hit “Clear all current trace data”.
  4. Execute the very last click in your application and wait until the error message dialogue appears.
  5. Manually refresh the trace website. You should only see 1 trace result that also contains your error message.
  6. Open the trace and click the button “jump to error”. You should see the first occurrence of a “Success = False” which can indicate where the bug originates.

Hi Joachim, translations is not a switch I can set this way. Translations seems like a default switch. It’s always enabled. But I can’t figure out how to disable it.

Hi Joachim, I used the process switch. But no error in the trace. So now I checked the process in Architect and compared it with the trace. Trace just stopped after plugin duplicatetreeaction couldn’t perform a duplicate of nothing. No error.

It can happen indeed that a trace stops abruptly due to a plugin error. In that case, no error message is shown. In the case from your screenshot, I think that the “duplicatetreeaction” does not handle NULL inputs properly. I will write a ticket for our plugin devs to properly handle this error case.

1 Like

Hi Joachim, translations is not a switch I can set this way. Translations seems like a default switch. It’s always enabled. But I can’t figure out how to disable it.

As discussed, this seems to be a specific bug/feature probably related to custom web.config settings or something similar and should be resolvable via a service request in the portal

1 Like