DIY - Kibana Dashboards with Application Data - Part 3

This post is part of a series: DIY - Kibana Dashboards with Application

The major parts of creating a search index (using application data) in Elasticsearch and creating Dashboards (and visualizations) in Kibana were completed in Part 1 and Part 2. This part will focus on the final steps of importing the Kibana Dashboards (and visualizations) into PeopleSoft, configuring and deploying them as PeopleSoft visualizations (tiles, related content, etc.).

Import Kibana Dashboards

In Part 2, we created two dashboards (Full View and Tile View) in Kibana. These dashboards can be imported into PeopleSoft from PeopleTools > Search Framework > Administration > Import Kibana Dashboards.

Configure the Dashboard(s) as a Tile

Once the Dashboards are successfully imported into PeopleSoft, we can configure them as a Tile (or related content) from PeopleTools > Search Framework > Administration > Kibana Visualizations.

Note: When we setup the dashboard as a Tile, PeopleTools will automatically create the Tile content reference for us as shown below.

Specify User Privileges

Once we configure the Kibana dashboard(s) as a Tile, we can specify the user privileges to secure access to the Tile.

PeopleTools > Search Framework > Administration > Kibana Privileges

Note: This will automatically update the Tile content reference security appropriately.

Deploy Kibana Dashboards

Finally, we can deploy (redeploy?) the dashboards configured in PeopleSoft as a visualization (Tile) into Kibana. This is step will deploy all the configurations (security, etc.) associated with the dashboards in Kibana.

PeopleTools > Search Framework > Administration > Deploy Kibana Dashboards

DIY - Kibana Dashboards with Application Data - Part 2

This post is part of a series: DIY - Kibana Dashboards with Application

Once the search index (based on application data) is deployed and functional in Elasticsearch, we can now move ahead with the process of configuring visualizations and dashboards in Kibana.

Create Index Pattern in Kibana

At this stage, we know that our custom search index is working in Elasticsearch. The next step is to create an index pattern in Kibana. This index pattern will then become the hook (connection) between the search index in Elasticsearch and the visualizations in Kibana.

Let us login to Kibana (using PS userid) and create the index pattern based on the search index in Elasticsearch.

Accessing Kibana Dashboards from PeopleSoft

PS userid has elevated security in the application so we know that we can login to Kibana without any issues using that userid. But what is the security required to be able to login to Kibana?

PeopleBooks: Accessing Kibana Dashboards from PeopleSoft

"A PeopleSoft user with the Search Administrator role or a user who has edit privilege or create privilege to any of the dashboard can log on to Kibana directly. The edit or create privilege is specified on the Kibana Privileges page."

After logging into Kibana as PS userid (or other userid with appropriate access), we can create the index pattern based on the search index as shown below.

Kibana > Management > Index Patterns > Create Index Pattern

Create Visualizations

Now that we have everything in place, we can start creating our visualizations in Kibana.

Tag Cloud Visualization

Horizontal Bar Visualization

Vertical Bar Visualization

Pie Visualization

Saved Search with Drilling URL

Before we create a saved search with drilling URL, we must change the format of the drilling URL field from string to URL. This will ensure that the URL is actually converted into a hyperlink in the visualization.

Kibana > Management > Index Patterns > Select Index Pattern: sv_ib_async_msg* > Field: ORCL_ES_LINK

Now we can go ahead and create a saved search.

Create Dashboards

The first dashboard created in the video shown below will be the detailed dashboard (Full View) which contains all the visualizations.

The next video demonstrates how to create a Dashboard with a single visualization. This will be used as the Tile Dynamic Content for the Kibana Visualizer (Tile View).

That is it! Our job within Kibana is completed.

DIY - Kibana Dashboards with Application Data - Part 1

This post is part of a series: DIY - Kibana Dashboards with Application

Create Search Definition

Create Search Query

PeopleBooks: Understanding Search Definition Queries

We will expose the required application data (PSAPMSGPUBHDR table in this example) as a search index by creating a query and referencing the query in the search definition.

From PeopleBooks: "The data returned by the query/connected query serves as the source data that the search engine crawls to create the search index."

The requirements to build a query (based on application data) are simple and shown below.

Create Search Definition

PeopleBooks: Creating Search Definitions

A simple search definition was created using the query created earlier. For the purpose of this post, we will not set any security on this search definition. Basically, the search definition will be accessible as long as we have access to the application.

Create Search Category

PeopleBooks: Understanding Search Categories

A simple search category was created referencing the search definition from the previous section. The 'Search Group' checkbox in the 'General' tab allows us to enable this search index via Global Search.

Deploy Search Definition

PeopleBooks: Administering Search Definitions and Search Categories

From PeopleBooks: "Before end users can run searches against the search indexes, the search definitions need to be deployed to the search engine so that the search engine can create the structure of the search index based on the search definition, crawl the defined search criteria, and populate the index with the results of the search definition query."

We simply need to navigate to PeopleTools > Search Framework > Administration > Deploy/Delete Object, select our search definition and 'Deploy'. The status of our search definition will change to 'Deployed' and 'Not Crawled'.

Schedule Search Index

Next we can schedule the search index build process from PeopleTools > Search Framework > Administration > Schedule Search Index.

Once PTSF_GENFEED process completes successfully, we should see the status of our search definition deployment change to 'Deployed' and 'Successful'.

PeopleTools > Search Framework > Administration > Deploy/Delete Object

Define Search Context

Our Search Category must be added to the 'Homepage' Search Context to enable our search to appear as part of Global Search.

PeopleTools > Search Framework > Administration > Define Search Context

Setup Search Group Security

Finally, we need to also ensure that the Search Group is added to an appropriate Permission List to enable access via Global Search.

Refer: E-ES/SES: Search Category not Showing Up in Global Search Bar (Doc ID 1534166.1)

Verify Search Index

We can verify that our search index is functional either using the 'Search Test Page' or using the Global Search as shown below.

DIY - Kibana Dashboards with Application Data

In my recent post, I wrote about getting started with Kibana dashboards in PeopleTools 8.58 using application data. In that post, we saw how we can deploy and use some of the delivered Kibana dashboards that were built using application data (search indices).

Refer: PeopleTools 8.58 - Application Data and Kibana Dashboards

While it is nice to use some of the delivered dashboards as examples to kick the tires, we don't have to wait for Oracle/PeopleSoft to deliver search indices (based on application data) and Kibana dashboards (based on those search indices) to take advantage of this functionality. The entire framework is available in PeopleTools! Why not build (configure) our own from scratch?

This post will provide details on how we can do just that.

Demonstration

The below video shows a completely custom Kibana Dashboard configured as a Tile and rendered using the Kibana Visualizer delivered in PeopleSoft.

In this example, I used Integration Broker asynchronous message data (PSAPMSGPUBHDR table) as the 'application data' use case.

Integration Broker > Service Operations Monitor > Monitoring > Asynchronous Services

Steps to Do It Yourself

PeopleBooks: Understanding Application Data and Kibana Dashboards

The PeopleBooks referenced above describes the process of creating visualizations in Kibana and making them available as tiles (or related content). Before we begin with the first step of creating visualizations and dashboards in Kibana, we must create and deploy a search definition based on the required application data, crawl the search index and ensure that it is available in Elasticsearch. This is required because the Kibana visualization will be built based on this application search index.The below posts detail the end to end process in three parts.

DIY - Kibana Dashboards with Application Data - Part 1

This post covers how we can create a search definition based on application data, deploy the search index on Elasticsearch and verify the search index functionality.

DIY - Kibana Dashboards with Application Data - Part 2

This post covers how we can create Visualizations and Dashboards in Kibana using PeopleSoft application data (search indices).

DIY - Kibana Dashboards with Application Data - Part 3

This post will cover how we can import Kibana Dashboards (and Visualizations) into PeopleSoft, configure and deploy them as PeopleSoft visualizations (tiles, related content, etc.).

Updated on June 12th, 2020

Here is a video of my presentation - DIY Kibana Dashboards at Education Innovation Sessions hosted by CY2 and partners.

https://www.youtube.com/watch?v=awiL3f7j4KI

Drop Zones for Classic

PeopleTools 8.58 introduced Drop Zones for Classic. I previously wrote about a SQL query that helps us identify a list of delivered components (both Fluid and Classic) that contain Drop Zones.

Refer: SQL to list Components with Drop Zones

At the time of writing (April 13, 2020), I reported that I could not find any application PUM image that contained classic components with Drop Zones. I tried executing the same SQL in a HCM - PUM image 34 (which is also on 8.58.03) and to my surprise I found 20 classic components with Drop Zones. Kudos to the HCM team for leveraging a lot of the latest PeopleTools features such as Kibana Dashboards, Classic Drop Zones, etc.!!

Oracle ACE Director, Graham Smith, showed us (in detail with a simple example) how we can use the Drop Zones that are delivered for Fluid components.

Refer: http://i-like-trains.blogspot.com/2019/04/drop-zones-and-hcm-image-030.html

Further details can be found in PeopleBooks: Understanding Drop Zones

COMPANY_TABLE is an example of a classic component in HCM Image 34 which contains Drop Zones.

Let us create a simple subpage and test the Drop Zone feature in Classic.

Custom subpage with page field elements

Let us create a custom subpage (as described in the Peoplebooks referenced earlier). This contains the required group box (container) that encompasses the entire subpage. All other page elements on this subpage must be enclosed within this container. We need to also clear the group box label text, set label style to PAGROUPBOXLABELINVISIBLE, select Adjust Layout for Hidden Fields check box and optionally select the Hide Border check box.

Configure Drop Zone

PeopleTools > Portal > Configure Drop Zones

Demo

PeopleTools 8.58 - Application Data and Kibana Dashboards

PeopleTools 8.58 has introduced the Kibana Visualizer within the PeopleSoft user interface. This is a feature that many of us were looking forward to using since we saw some demos at Oracle OpenWorld last year.

I recently installed HCM PUM Image 34 on my PC as a VirtualBox instance. While I knew it was on PeopleTools 8.58.03, I was pleasantly surprised that the instance was packaged (DPK) with Elasticsearch 7.0 and Kibana 7.0.

Per PeopleBooks referenced below, we need to ensure a few things before getting started with Kibana Dashboards in PeopleSoft.

PeopleBooks: Understanding Application Data and Kibana Dashboards

We already know from the VirtualBox manifest that we are on PeopleTools 8.58.03, Elasticsearch 7.0 and Kibana 7.0.

But before we get started, how do we ensure that both Elasticsearch and Kibana are running? I was getting inconsistent results on my VBox image where they would appear to be running after the initial install but will not start up after a restart.

A big thank you to my friend Andrew Casey who helped me with the commands to start Elastic and Kibana.

Start up Elasticsearch

Let us assume that elasticsearch is installed but not running. We need to first make sure that we can execute java within the Elasticsearch directory from command line as esadm1 user.

Next we need to start up elasticsearch using nohup ./elasticsearch & command.

Refer: E-ES: How to Start and Stop the Elasticsearch Service on Windows and Linux Operating Systems (Doc ID 2404285.1)

Verify Elasticsearch is running

We can easily verify if Elasticsearch is functional using several methods. We can navigate to PeopleTools > Search Framework > Administration > Search Instance. Then we can Ping the search instance PTSF_DEFAULT. Alternatively, we can simply perform a global search and verify the results.

Start Up Kibana

Now that Elasticsearch is running, we can turn our attention to Kibana. If Kibana has not been started, we will see the following error when we navigation to any delivered Kibana visualization/dashboard. For example, Recruiting (Homepage) > Application Details (Tile).

We can start Kibana by logging into putty as esadm1 and executing the nohup ./kibana & command.

Verify Kibana is running

Once we start Kibana, we can verify that Kibana is functional by logging into Kibana directly (using port 5601 in the case of VirtualBox) or by accessing a delivered visualization/dashboard.

Now we know that Kibana is working fine even though we see the following message.

This message only means that we don't have any index patterns created. We will address this in the following sections.

Getting started with Application Data and Kibana Dashboards

Now that we know how to consistently ensure that both Elasticseearch and Kibana are functional on our demo PUM image, we can start playing with some of the delivered (HCM PI 34) Kibana dashboards and visualizations.

From the PeopleBooks reference shared earlier, here is the process to get started with Kibana dashboards and visualizations.

In the case of delivered Kibana dashboards and visualizations, steps 1 and 2 are already completed by Oracle. The following delivered dashboards were available under PeopleTools > Search Framework > Administration > Import Kibana Dashboards.

Let us focus on the 'Application Details' dashboard in this blog post. We can see this dashboard has been configured as a Tile under PeopleTools > Search Framework > Administration > Kibana Visualizations.

We can also see how this Tile content reference is configured under PeopleTools > Portal > Structure and Content - Fluid Structure and Content > Fluid Pages > Analytics > Application Details. This covers step 3 from PeopleBooks.

The visualization security is configured under PeopleTools > Search Framework > Administration > Kibana Privileges. This covers step 4.

Step 5 requires us to deploy the dashboard that was configured in PeopleTools to Kibana. We can do this from PeopleTools > Search Framework > Administration > Deploy Kibana Dashboards. We can see from the image below that the status is 'Undeployed'. Let us go ahead and deploy the dashboard.

Since PS user already has access to all the roles assigned to this dashboard, we can now go ahead and test the dashboard.

Testing

While we continue to receive the same message as before, we will now notice that there are index patterns created for us by the deployment process.

Let us complete the configuration by selecting the delivered hc_hrs_app_index index pattern and setting it as the default as shown in the video below.

Wait! Does this mean all users need to do this setup? No. This is a one time configuration that must be completed in Kibana.

Demo