Kibana — Log Visualization Tool

Overview

This page covers our internal log visualisation tool, powered by open-source Kibana, to help Support and SA/IE teams debug production issues. The aim is to make log analysis accessible to non-technical users by offering clear instructions and examples.

Prerequisites

While using Kibana doesn't require any prerequisite, a basic understanding of Lucene / KQL queries is helpful for advanced filtering.

Purpose & use case

• Allow a new user of Kibana to find the logs that could be helpful for a given troubleshooting.
• Build an understanding of the different kinds of logs that HyperVerge applications publish, for advanced debugging.

Log types and index names

Our system generates various types of logs, each stored in specific indices:

1. Request Response Logs — contain request payloads (body and headers), response bodies, status codes, processing times, content lengths, etc. Useful for understanding individual request details.

   • Index name: kubera-log-request-*
   • Retention: 90 days

2. Application Logs — generated by backend applications, essential for advanced debugging.

   • Index name: kubera-log-app-*
   • Retention: 7 days

3. Nginx Logs — produced by our Nginx reverse proxy at the backend application's entry point. Include a subset of data from request response logs. For most services, nginx.request_time also accounts for the time taken to upload the request payload from the client application.

   • Index name: filebeat-*
   • Retention: 7 days

4. System Logs (syslogs) — insights into system or OS-level events.

   • Index name: filebeat-*
   • Retention: 7 days

5. Machine Access Logs — details of users who have accessed the machine/system/server.

   • Index name: filebeat-*
   • Retention: 7 days

tip

The index name selector is at the top-left of the Discover page of Kibana, under the search bar.

Standard fields across logs

Several fields are consistent across different log types:

• transactionId — in req.headers.transactionid (request response logs) and nginx.req_headers.transactionId (Nginx logs).
• appId — in req.headers.appid (request response logs) and nginx.req_headers.appId (Nginx logs). May be absent in Nginx logs if not in request headers (e.g., when using Authorization Token for authentication).
• requestId — requestId in request response logs, nginx.res_headers.x-request-id in Nginx logs, requestId in application logs.
• product — the product or service associated with the log, accessible via the product keyword in request response and application logs.
• region — AWS or other cloud region where the request was processed. cloud.region in request response/application logs; meta.cloud.region in Nginx logs.
• url — the API endpoint: nginx.access.url in Nginx logs, req.originalUrl in request response logs.

Commonly used fields per index

Request Response Logs (kubera-log-request-*)

• req.headers.transactionid
• req.headers.appid
• requestId
• product
• beat.name
• cloud.region
• res.statusCode
• responseTime
• req.originalUrl
• req.headersStr
• req.bodyStr
• res.bodyStr

App Logs (kubera-log-app-*)

• product
• cloud.region
• requestId
• logline
• level
• message
• beat.name

Nginx Logs (filebeat-*)

• nginx.res_headers.x-request-id
• nginx.req_headers.transactionId
• nginx.req_headers.appId
• nginx.status
• nginx.upstream_status
• nginx.request_time
• nginx.upstream_response_time
• nginx.proxy_add_x_forwarded_for OR nginx.remote_ip
• beat.name

Using Kibana for log analysis

Accessing Kibana

1. Open Kibana at https://cg-1.log.hyperverge.org/.
2. Log in with your HyperVerge Google account.

Discovering logs

1. Navigate to Discover at https://cg-1.log.hyperverge.org/app/kibana#/discover.
2. Select the index pattern from the dropdown menu (e.g., kubera-log-request-*).
3. Set the time range in the top-right corner.
   warning

   Keep the duration as specific as possible and do not use a time window greater than a week for faster processing and to reduce load on the Elastic cluster.

4. Search logs — enter search queries in the search bar to filter logs based on specific criteria. You can also click Add a filter to filter by specific fields. Multiple fields can be combined.

TODO

Screenshots: Discover page, index selector, search bar, and "Add a filter" panel. (Pending migration from Notion.)

Example — trace a specific request

Trace a specific request via transactionId

1. In the search bar, enter: req.headers.transactionid: "YOUR_TRANSACTION_ID".
2. View results — Kibana displays all logs associated with the specified transactionId in the given time frame.

Find a specific request in Nginx and App Logs (advanced debugging)

1. Get the RequestId from request-response logs

   1. Filter for req.originalUrl: "/v1/readId" AND res.statusCode: 500. Use a longer time frame if no results show up.
   2. Find a sample request and copy the value for requestId.

2. Find the request in Nginx logs

   1. Open Kibana in a new tab and select filebeat-* index. Use the same time frame.
   2. Filter for nginx.res_headers.x-request-id: <requestId>.

3. Find the request in App Logs

   1. Open Kibana in a new tab and select kubera-log-app-* index. Use the same time frame.
   2. Filter for requestId: <requestId>.
   3. Read through the log lines / messages to identify the cause of the 500 error.

Advanced — KQL queries

For more refined searches, use Kibana Query Language (KQL) / Lucene syntax:

Basic syntax:

• Specific status code: req.statusCode: 200
• Range of status codes: req.statusCode: [400 TO 499]

Combining conditions:

• req.originalUrl: "/v1/readId" AND res.statusCode: 200

Wildcards and regex:

• URLs starting with /api: req.originalUrl: "/api*"
• Regex: res.bodyStr: /error.*/

For a full KQL tutorial, see the official Elastic documentation: Kibana Query Language.

Additional resources

• KQL reference
• Kibana visualisations tutorial