Regular Expression Pattern Matching for PeopleSoft Trace/Log Files

Have you ever wondered how to remove some of the timing and other environment specific information from a trace/log file so that we only see the SQL and PeopleCode?

The timing and other information that I am referring to are:

SQL:

PeopleCode:

Why?

While these timing and other information in the trace file are great for performance tuning activities, they seem to cause several issues (at least for me) when doing a comparative analysis. By comparative analysis, I mean comparing the trace files across different scenarios to identify a problem. For example, an issue might only occur for user A but does not occur for user B. Then we would be inclined towards comparing the trace files generated for user A and user B in an effort to identify the problem. Similarly, an issue might only occur in a particular environment (say TEST) whereas it works fine in another environment (say DEVELOPMENT). Again we would be inclined towards comparing the trace files generated in TEST and DEVELOPMENT to identify the problem.

Now that I detailed the scenario/activity, I would like to describe the problem. When we try to compare two different trace files using any file compare tools (such as WinMerge), the results would identify a difference on almost every single line! Most of them would be false positives due to the timing and other information which is not really useful in such cases.

E.g.:

Regular Expressions (RegEx) Pattern Matching for PeopleSoft Trace/Log Files:

So, if we get rid of (or mask the unncessary information) then we can determine the "actual" differences more efficiently.

That is where Regular Expression (RegEx) pattern matching helps in identifying and removing/masking unncessary information.

Note: I generally use notepad++ for text editing. I am sure most text editors would have a similar RegEx find/replace functionality.

RegEx Pattern for SQL:

Find: ^[a-z]{8}\.\d+ +\(\d+\) +\t +\d+-\d+ +\d{2}\.\d{2}\.\d{2} +\d+\.\d{6} Cur#\d+\.\d+\.[a-zA-Z0-9_]+ RC=\d+ Dur=\d+.\d{6}
Replace: SQL---------------

RegEx Pattern for PeopleCode: 

Find: ^[a-z]{8}\.\d+ +\(\d+\) +\t +\d+-\d+ +\d{2}\.\d{2}\.\d{2} +\d+.\d{6}
Replace: PeopleCode--------

Benefits of Regex Pattern Matching Find/Replace During Comparative Analysis:

Now, if we remove all the unnecessary information and then compare, we would be able to determine the differences quickly and easily. Here is an example of how I used the RegEx Find/Replace (detailed above) and then compared two different trace files using WinMerge.

Similarly, we can use Regex Pattern Matching to intelligently find and replace specific portions of the trace/log files to improve readability!

PeopleSoft - Mining Linux based Web/App Server Logs Using find, egrep, awk commands

This post is based on information provided to me by an ex-colleague, friend and the best PS Admin I have worked with Danny Kile. I use this regularly and find it very effective when we need to quickly investigate issues particularly in a Production Environment (with several instances of the web/app server domains).

# Change Directory to where the logs reside
# Note: I changed the directory to the appserv folder so the search would go through
# multiple domain folders (if they exist)
cd /%PS_CFG_HOME%/appserv/

# If you want all the output printed
find . -name 'APPSRV_1014.LOG' | xargs egrep '*Search Text*'

# If you just want the number of times the 'Search Text' was found
find . -name 'APPSRV_1014.LOG' | xargs egrep '*Search Text*' | wc -l

# If you want multiple days worth
find . -name 'APPSRV_101*.LOG' | xargs egrep '*Search Text*' | wc -l

#If you want to aggregate based on day or hour, etc.
find . -name 'APPSRV_101*.LOG' | xargs awk '/*Search Text*/{freq[$3]++} END {printf "\n";for(day in freq){printf "%s\t%d\n",day,freq[day];}printf "\n";printf "\n";}'

Hope this is useful. Please share any other commands/scripts that might come in handy for PeopleSoft Developers.