Kibana Visualizations: Working with Region Maps

By now, we all must have seen the demonstrations of Kibana Data Visualizer that was made available in PeopleTools 8.58. If not, it is very easy to spin up a PUM image (HCM or FSCM) and get a hands-on experience with this functionality.

While evaluating the delivered Kibana Visualizations, I noticed a few map visualizations.

For example, HCM > Workforce Administrator (Homepage) > Workforce Insight (Dashboard) > Workforce Equity (Tile) > Employee Distribution by Country (Visualization)

How does this work?

As we can see in the footer, the map is rendered to us using the Elastic Maps Service (EMS) which in turn produces basemap tiles leveraging OpenStreetMap data.

We can find more information here:
https://www.elastic.co/elastic-maps-service
https://maps.elastic.co

Further, the out-of-box Kibana functionality would directly request the maps from the browser. The EMS requests are made to the following domains:

• tiles.maps.elastic.co
  
• vectors.maps.elastic.co
  

If we run into firewall restrictions and unable to allow access to *.maps.elastic.co, we have the option of hosting the EMS locally.

We can find more information here:
https://www.elastic.co/guide/en/kibana/current/maps-connect-to-ems.html

Gotcha

When I dug in a little further to review how the application data (PS Query/Connected Query > Elasticsearch Index > Kibana Index Pattern) is hooked (or joined) to the EMS, I found a strange Kibana bug (?).

Here is the 'Data' tab of the Visualization configuration in Kibana. We can see that the COUNTRY field in PeopleSoft is being used as the aggregation field.

Now, if we go to the 'Options' tab, we will see that the Join Field for the EMS Vector map (World Countries) is set to 'ISO 3166-1 alpha-2 code'.

Wait a minute. The COUNTRY field in PeopleSoft is the 3 character ISO code but somehow it is joined with the 2 character ISO code. How is this possible? Am I right?

We can even preview on EMS to see the codes. If you don't believe me.

After scratching my head for a long time, I found that this is a bug with respect to the display of the Join Field in the Kibana configuration. It possibly may be defaulting to the first value that is in the drop-down. However, once I reviewed the actual JSON configuration (Saved Object) of the visualization, I found that it is in fact using the 'ISO 3166-1 alpha-3 code'. So, it proves that I was not losing my mind (at least as yet!). 😅

Exploring

Next, I wanted to build a map on my own. I used the same data 'Employee Distribution' and represented it per Country and State instead of just the Country.

I created an expression in my PS Query (used in Connected Query source for Search Definition) that concatenates COUNTRY_2CHAR and STATE fields into a single query field.

Next, I created a visualization in Kibana using this field as the Join Field for an 'Administrative region' vector map - 'Region ISO code'.

I look forward to your questions, comments and feedback (please use the comments section below).

Migrating Custom Search Definition and Configuration

In the following DIY blog series, I described how we can use the framework provided in PeopleTools 8.58 to configure our own custom visualizations and dashboards in Kibana and use them in PeopleSoft applications via the 'Kibana Visualizer'.

Refer: DIY - Kibana Dashboards with Application Data

Someone ask a very valid question:
How do we migrate these configurations from one environment (source) to another (target)?

This post will detail the steps for migrating the Search Definition and associated configuration.

Migration

We can migrate everything related to the Search Definition using an App Designer project. Simply insert the Search Definition and make sure the the 'Related Definitions' are selected as shown below.

Save the 'Project' and migrate to target environment.

Note: If we used any Application Package PeopleCode to configure the Search Definition (Security Configuration) - Document Filter App Class, then we must also include the App Package in the project.

Post Migration Steps

Once we migrate the configuration, we must complete the following administrative steps in the target environment.

1. Deploy Search Definition
2. Schedule Search Index
3. Configure Search Context (optional)
4. Configure Search Group Security

Related Posts

Migrating Custom Kibana Visualizations/Dashboards

DIY - Kibana Dashboards with Application Data - Part 3

This post is part of a series: DIY - Kibana Dashboards with Application

The major parts of creating a search index (using application data) in Elasticsearch and creating Dashboards (and visualizations) in Kibana were completed in Part 1 and Part 2. This part will focus on the final steps of importing the Kibana Dashboards (and visualizations) into PeopleSoft, configuring and deploying them as PeopleSoft visualizations (tiles, related content, etc.).

Import Kibana Dashboards

In Part 2, we created two dashboards (Full View and Tile View) in Kibana. These dashboards can be imported into PeopleSoft from PeopleTools > Search Framework > Administration > Import Kibana Dashboards.

Configure the Dashboard(s) as a Tile

Once the Dashboards are successfully imported into PeopleSoft, we can configure them as a Tile (or related content) from PeopleTools > Search Framework > Administration > Kibana Visualizations.

Note: When we setup the dashboard as a Tile, PeopleTools will automatically create the Tile content reference for us as shown below.

Specify User Privileges

Once we configure the Kibana dashboard(s) as a Tile, we can specify the user privileges to secure access to the Tile.

PeopleTools > Search Framework > Administration > Kibana Privileges

Note: This will automatically update the Tile content reference security appropriately.

Deploy Kibana Dashboards

Finally, we can deploy (redeploy?) the dashboards configured in PeopleSoft as a visualization (Tile) into Kibana. This is step will deploy all the configurations (security, etc.) associated with the dashboards in Kibana.

PeopleTools > Search Framework > Administration > Deploy Kibana Dashboards

DIY - Kibana Dashboards with Application Data - Part 2

This post is part of a series: DIY - Kibana Dashboards with Application

Once the search index (based on application data) is deployed and functional in Elasticsearch, we can now move ahead with the process of configuring visualizations and dashboards in Kibana.

Create Index Pattern in Kibana

At this stage, we know that our custom search index is working in Elasticsearch. The next step is to create an index pattern in Kibana. This index pattern will then become the hook (connection) between the search index in Elasticsearch and the visualizations in Kibana.

Let us login to Kibana (using PS userid) and create the index pattern based on the search index in Elasticsearch.

Accessing Kibana Dashboards from PeopleSoft

PS userid has elevated security in the application so we know that we can login to Kibana without any issues using that userid. But what is the security required to be able to login to Kibana?

PeopleBooks: Accessing Kibana Dashboards from PeopleSoft

"A PeopleSoft user with the Search Administrator role or a user who has edit privilege or create privilege to any of the dashboard can log on to Kibana directly. The edit or create privilege is specified on the Kibana Privileges page."

After logging into Kibana as PS userid (or other userid with appropriate access), we can create the index pattern based on the search index as shown below.

Kibana > Management > Index Patterns > Create Index Pattern

Create Visualizations

Now that we have everything in place, we can start creating our visualizations in Kibana.

Tag Cloud Visualization

Horizontal Bar Visualization

Vertical Bar Visualization

Pie Visualization

Saved Search with Drilling URL

Before we create a saved search with drilling URL, we must change the format of the drilling URL field from string to URL. This will ensure that the URL is actually converted into a hyperlink in the visualization.

Kibana > Management > Index Patterns > Select Index Pattern: sv_ib_async_msg* > Field: ORCL_ES_LINK

Now we can go ahead and create a saved search.

Create Dashboards

The first dashboard created in the video shown below will be the detailed dashboard (Full View) which contains all the visualizations.

The next video demonstrates how to create a Dashboard with a single visualization. This will be used as the Tile Dynamic Content for the Kibana Visualizer (Tile View).

That is it! Our job within Kibana is completed.

DIY - Kibana Dashboards with Application Data - Part 1

This post is part of a series: DIY - Kibana Dashboards with Application

Create Search Definition

Create Search Query

PeopleBooks: Understanding Search Definition Queries

We will expose the required application data (PSAPMSGPUBHDR table in this example) as a search index by creating a query and referencing the query in the search definition.

From PeopleBooks: "The data returned by the query/connected query serves as the source data that the search engine crawls to create the search index."

The requirements to build a query (based on application data) are simple and shown below.

Create Search Definition

PeopleBooks: Creating Search Definitions

A simple search definition was created using the query created earlier. For the purpose of this post, we will not set any security on this search definition. Basically, the search definition will be accessible as long as we have access to the application.

Create Search Category

PeopleBooks: Understanding Search Categories

A simple search category was created referencing the search definition from the previous section. The 'Search Group' checkbox in the 'General' tab allows us to enable this search index via Global Search.

Deploy Search Definition

PeopleBooks: Administering Search Definitions and Search Categories

From PeopleBooks: "Before end users can run searches against the search indexes, the search definitions need to be deployed to the search engine so that the search engine can create the structure of the search index based on the search definition, crawl the defined search criteria, and populate the index with the results of the search definition query."

We simply need to navigate to PeopleTools > Search Framework > Administration > Deploy/Delete Object, select our search definition and 'Deploy'. The status of our search definition will change to 'Deployed' and 'Not Crawled'.

Schedule Search Index

Next we can schedule the search index build process from PeopleTools > Search Framework > Administration > Schedule Search Index.

Once PTSF_GENFEED process completes successfully, we should see the status of our search definition deployment change to 'Deployed' and 'Successful'.

PeopleTools > Search Framework > Administration > Deploy/Delete Object

Define Search Context

Our Search Category must be added to the 'Homepage' Search Context to enable our search to appear as part of Global Search.

PeopleTools > Search Framework > Administration > Define Search Context

Setup Search Group Security

Finally, we need to also ensure that the Search Group is added to an appropriate Permission List to enable access via Global Search.

Refer: E-ES/SES: Search Category not Showing Up in Global Search Bar (Doc ID 1534166.1)

Verify Search Index

We can verify that our search index is functional either using the 'Search Test Page' or using the Global Search as shown below.

DIY - Kibana Dashboards with Application Data

In my recent post, I wrote about getting started with Kibana dashboards in PeopleTools 8.58 using application data. In that post, we saw how we can deploy and use some of the delivered Kibana dashboards that were built using application data (search indices).

Refer: PeopleTools 8.58 - Application Data and Kibana Dashboards

While it is nice to use some of the delivered dashboards as examples to kick the tires, we don't have to wait for Oracle/PeopleSoft to deliver search indices (based on application data) and Kibana dashboards (based on those search indices) to take advantage of this functionality. The entire framework is available in PeopleTools! Why not build (configure) our own from scratch?

This post will provide details on how we can do just that.

Demonstration

The below video shows a completely custom Kibana Dashboard configured as a Tile and rendered using the Kibana Visualizer delivered in PeopleSoft.

In this example, I used Integration Broker asynchronous message data (PSAPMSGPUBHDR table) as the 'application data' use case.

Integration Broker > Service Operations Monitor > Monitoring > Asynchronous Services

Steps to Do It Yourself

PeopleBooks: Understanding Application Data and Kibana Dashboards

The PeopleBooks referenced above describes the process of creating visualizations in Kibana and making them available as tiles (or related content). Before we begin with the first step of creating visualizations and dashboards in Kibana, we must create and deploy a search definition based on the required application data, crawl the search index and ensure that it is available in Elasticsearch. This is required because the Kibana visualization will be built based on this application search index.The below posts detail the end to end process in three parts.

DIY - Kibana Dashboards with Application Data - Part 1

This post covers how we can create a search definition based on application data, deploy the search index on Elasticsearch and verify the search index functionality.

DIY - Kibana Dashboards with Application Data - Part 2

This post covers how we can create Visualizations and Dashboards in Kibana using PeopleSoft application data (search indices).

DIY - Kibana Dashboards with Application Data - Part 3

This post will cover how we can import Kibana Dashboards (and Visualizations) into PeopleSoft, configure and deploy them as PeopleSoft visualizations (tiles, related content, etc.).

Updated on June 12th, 2020

Here is a video of my presentation - DIY Kibana Dashboards at Education Innovation Sessions hosted by CY2 and partners.

https://www.youtube.com/watch?v=awiL3f7j4KI