Kibana Visualizations: Working with Region Maps

By now, we all must have seen the demonstrations of Kibana Data Visualizer that was made available in PeopleTools 8.58. If not, it is very easy to spin up a PUM image (HCM or FSCM) and get a hands-on experience with this functionality.

While evaluating the delivered Kibana Visualizations, I noticed a few map visualizations.

For example, HCM > Workforce Administrator (Homepage) > Workforce Insight (Dashboard) > Workforce Equity (Tile) > Employee Distribution by Country (Visualization)

How does this work?

As we can see in the footer, the map is rendered to us using the Elastic Maps Service (EMS) which in turn produces basemap tiles leveraging OpenStreetMap data.

We can find more information here:
https://www.elastic.co/elastic-maps-service
https://maps.elastic.co

Further, the out-of-box Kibana functionality would directly request the maps from the browser. The EMS requests are made to the following domains:

• tiles.maps.elastic.co
  
• vectors.maps.elastic.co
  

If we run into firewall restrictions and unable to allow access to *.maps.elastic.co, we have the option of hosting the EMS locally.

We can find more information here:
https://www.elastic.co/guide/en/kibana/current/maps-connect-to-ems.html

Gotcha

When I dug in a little further to review how the application data (PS Query/Connected Query > Elasticsearch Index > Kibana Index Pattern) is hooked (or joined) to the EMS, I found a strange Kibana bug (?).

Here is the 'Data' tab of the Visualization configuration in Kibana. We can see that the COUNTRY field in PeopleSoft is being used as the aggregation field.

Now, if we go to the 'Options' tab, we will see that the Join Field for the EMS Vector map (World Countries) is set to 'ISO 3166-1 alpha-2 code'.

Wait a minute. The COUNTRY field in PeopleSoft is the 3 character ISO code but somehow it is joined with the 2 character ISO code. How is this possible? Am I right?

We can even preview on EMS to see the codes. If you don't believe me.

After scratching my head for a long time, I found that this is a bug with respect to the display of the Join Field in the Kibana configuration. It possibly may be defaulting to the first value that is in the drop-down. However, once I reviewed the actual JSON configuration (Saved Object) of the visualization, I found that it is in fact using the 'ISO 3166-1 alpha-3 code'. So, it proves that I was not losing my mind (at least as yet!). 😅

Exploring

Next, I wanted to build a map on my own. I used the same data 'Employee Distribution' and represented it per Country and State instead of just the Country.

I created an expression in my PS Query (used in Connected Query source for Search Definition) that concatenates COUNTRY_2CHAR and STATE fields into a single query field.

Next, I created a visualization in Kibana using this field as the Join Field for an 'Administrative region' vector map - 'Region ISO code'.

I look forward to your questions, comments and feedback (please use the comments section below).

Migrating Custom Kibana Visualizations/Dashboards

In the following DIY blog series, I described how we can use the framework provided in PeopleTools 8.58 to configure our own custom visualizations and dashboards in Kibana and use them in PeopleSoft applications via the 'Kibana Visualizer'.

Refer: DIY - Kibana Dashboards with Application Data

Someone ask a very valid question:
How do we migrate these configurations from one environment (source) to another (target)?

My previous blog post detailed the steps for migrating the Search Definition and associated configuration. This post will describe the steps for migrating the Kibana Visualizations and Dashboards.

Refer: Migrating Custom Search Definition and Configuration

Migration

We used an App Designer project to migrate the Search Definition. We will be using Data Migration Workbench to migrate all the Kibana Visualization/Dashboard related configurations. The following Doc Id from My Oracle Support details the application data sets delivered by Oracle.

E-KB: How to Migrate Kibana Dashboards From One Environment to Another? (Doc ID 2684543.1)

Application Data Set	Description	Notes
PTSF_KIBANA_DBOARD_DTL	Defines the dashboards that are deployed to Kibana.	This is the configuration located in PeopleTools > Search Framework > Administration > Deploy Kibana Dashboards
PTSF_KIBANA_DBOARD_ATT	Defines the JSON files of a dashboard.
PTSF_KIBANA_DBOARD_ROLE_MAP	Defines the roles mapped to a dashboard.	This is the configuration located in PeopleTools > Search Framework > Administration > Kibana Privileges
PTSF_KIBANA_DBOARD_CREATE_ROLE	Defines the privileges assigned to a role with respect to a Kibana dashboard.
PTSF_KIBANA_SYSDB	Defines the system monitoring dashboards that are imported and deployed.	This is the configuration located in PeopleTools > Search Framework > Administration > Kibana Privileges We did not set this up in the examples demonstrated in the DIY series.

In addition to the above configurations, we also need to migrate any Tiles or Related Information configured using PeopleTools > Search Framework > Administration > Kibana Visualizations.

Application Data Set	Description	Notes
PSPRSM_CREF	Content Reference	Dashboards configured as Tiles.
RCF_SERVICES	RC Services	Dashboards configured as Related Information of a Fluid page.

Finally, if we used the following workaround to setup any Kibana dashboards as related content of classic pages, then we need to migrate the configuration using the data sets listed below.

Refer: Kibana Dashboards - Related Content for Classic

Application Data Set	Description	Notes
RCF_SERVICES	RC Services	Dashboards configured as Related Content of a Classic page.
RCF_SERVICE_DEFINITION	RC Service Definition	Custom Related Content Service Definition - Classic Kibana Related Content

Here is a project used for demonstration purposes. This project will be used to demonstrate the migration of all the configurations created in the DIY series.

Imported Kibana Dashboards

The following steps will migrate the dashboards imported into PeopleSoft from Kibana.

Demo

Kibana Privileges

Demo

Kibana Visualization as a Tile

Demo

Kibana Visualization as Related Information (Fluid)

Demo

Kibana Visualization as Related Content (Classic)

Demo

Post Migration Steps

Once we migrate all the data sets using Data Migration Workbench, we should deploy the imported dashboards to Kibana in the target environment. As you can see in the screenshot below, the imported dashboards will appear to be in deployed status. This status is brought forward from the source. Therefore, we must deploy once again in the target environment.

The deploy step takes care of all the configurations in Kibana - index patterns, visualizations and dashboards!

Migrating Custom Search Definition and Configuration

In the following DIY blog series, I described how we can use the framework provided in PeopleTools 8.58 to configure our own custom visualizations and dashboards in Kibana and use them in PeopleSoft applications via the 'Kibana Visualizer'.

Refer: DIY - Kibana Dashboards with Application Data

Someone ask a very valid question:
How do we migrate these configurations from one environment (source) to another (target)?

This post will detail the steps for migrating the Search Definition and associated configuration.

Migration

We can migrate everything related to the Search Definition using an App Designer project. Simply insert the Search Definition and make sure the the 'Related Definitions' are selected as shown below.

Save the 'Project' and migrate to target environment.

Note: If we used any Application Package PeopleCode to configure the Search Definition (Security Configuration) - Document Filter App Class, then we must also include the App Package in the project.

Post Migration Steps

Once we migrate the configuration, we must complete the following administrative steps in the target environment.

1. Deploy Search Definition
2. Schedule Search Index
3. Configure Search Context (optional)
4. Configure Search Group Security

Related Posts

Migrating Custom Kibana Visualizations/Dashboards

DIY - Kibana Dashboards with Application Data - Part 3

This post is part of a series: DIY - Kibana Dashboards with Application

The major parts of creating a search index (using application data) in Elasticsearch and creating Dashboards (and visualizations) in Kibana were completed in Part 1 and Part 2. This part will focus on the final steps of importing the Kibana Dashboards (and visualizations) into PeopleSoft, configuring and deploying them as PeopleSoft visualizations (tiles, related content, etc.).

Import Kibana Dashboards

In Part 2, we created two dashboards (Full View and Tile View) in Kibana. These dashboards can be imported into PeopleSoft from PeopleTools > Search Framework > Administration > Import Kibana Dashboards.

Configure the Dashboard(s) as a Tile

Once the Dashboards are successfully imported into PeopleSoft, we can configure them as a Tile (or related content) from PeopleTools > Search Framework > Administration > Kibana Visualizations.

Note: When we setup the dashboard as a Tile, PeopleTools will automatically create the Tile content reference for us as shown below.

Specify User Privileges

Once we configure the Kibana dashboard(s) as a Tile, we can specify the user privileges to secure access to the Tile.

PeopleTools > Search Framework > Administration > Kibana Privileges

Note: This will automatically update the Tile content reference security appropriately.

Deploy Kibana Dashboards

Finally, we can deploy (redeploy?) the dashboards configured in PeopleSoft as a visualization (Tile) into Kibana. This is step will deploy all the configurations (security, etc.) associated with the dashboards in Kibana.

PeopleTools > Search Framework > Administration > Deploy Kibana Dashboards

DIY - Kibana Dashboards with Application Data - Part 2

This post is part of a series: DIY - Kibana Dashboards with Application

Once the search index (based on application data) is deployed and functional in Elasticsearch, we can now move ahead with the process of configuring visualizations and dashboards in Kibana.

Create Index Pattern in Kibana

At this stage, we know that our custom search index is working in Elasticsearch. The next step is to create an index pattern in Kibana. This index pattern will then become the hook (connection) between the search index in Elasticsearch and the visualizations in Kibana.

Let us login to Kibana (using PS userid) and create the index pattern based on the search index in Elasticsearch.

Accessing Kibana Dashboards from PeopleSoft

PS userid has elevated security in the application so we know that we can login to Kibana without any issues using that userid. But what is the security required to be able to login to Kibana?

PeopleBooks: Accessing Kibana Dashboards from PeopleSoft

"A PeopleSoft user with the Search Administrator role or a user who has edit privilege or create privilege to any of the dashboard can log on to Kibana directly. The edit or create privilege is specified on the Kibana Privileges page."

After logging into Kibana as PS userid (or other userid with appropriate access), we can create the index pattern based on the search index as shown below.

Kibana > Management > Index Patterns > Create Index Pattern

Create Visualizations

Now that we have everything in place, we can start creating our visualizations in Kibana.

Tag Cloud Visualization

Horizontal Bar Visualization

Vertical Bar Visualization

Pie Visualization

Saved Search with Drilling URL

Before we create a saved search with drilling URL, we must change the format of the drilling URL field from string to URL. This will ensure that the URL is actually converted into a hyperlink in the visualization.

Kibana > Management > Index Patterns > Select Index Pattern: sv_ib_async_msg* > Field: ORCL_ES_LINK

Now we can go ahead and create a saved search.

Create Dashboards

The first dashboard created in the video shown below will be the detailed dashboard (Full View) which contains all the visualizations.

The next video demonstrates how to create a Dashboard with a single visualization. This will be used as the Tile Dynamic Content for the Kibana Visualizer (Tile View).

That is it! Our job within Kibana is completed.