This blog will explain how to debug a background batch program with real background mode (SY-BATCH is X).
Questions that will be answered are:
How to debug a running job?
How to debug a completed job?
Starting the debug mode for batch job
First we plan a single run of the batch job. In our example we run program RSWAITSEC which does nothing more than wait.
In SM37 show the job run:
Select the job and in the command line enter the background debug command JDBG:
Now the debugger starts first in the batch job part. Hit F7 a few time (F7 = jump back out of routine) until you reach the real program:
As you can see here the SY-BATCH variable is X, which means you are debugging with real background mode on.
SM50 background debugging
If you want to debug a running job, you have to goto SM50 and select the background process. Then choose the menu option Administration / Program / Debugging. Confirm the prompt:
Wait until the running SQL statement has completed and debug mode will start.
Please be very careful with this kind of debugging in a productive system. If you cancel the debug session there might be a rollback work statement triggered, which can cause database inconsistencies.
If you want to prevent same job from starting, while a previous instance is still running, you need to add program RSBTONEJOB or RSBTONEJOB2 as first step in the batch job. This step will detect if the previous instance is still running and abort to avoid the next steps from being executed. More background in OSS note 557610. The scope of both programs is in principle limited to be used for idocs and CUA. To extend the scope or in case of issues, read the instructions in note 3225033 – Behavior of program RSBTONEJOB, RSBTONEJOB2 not as expected.
Forcing a batch program to wait
If for some reason you need a batch program to wait between steps, you can use program RSWAITSEC as a step. This program will only do a wait for x amount of seconds.
Setting current date in the TVARVC table for the RV variables
The RV variables for current date, month and year are often used in month end closing batch jobs. Running program RVSETDAT will set the current dates for RV TVARC variables.
Checking basis background job system settings and working
If you want to validate if the background job system function itself is working properly, start transaction SM65 background processing analysis tool.
Result is shown correct working, number of batches and wait time:
Start transaction ST13 and start tool BACKGROUND_JOB_ANALYSIS. Or directly start program /SSA/BTC.
Pending on your selection you get a graphical overview or a full list for you to speed up your analysis.
Mass stop and start of batch jobs
Programs BTCTRNS1 and BTCTRNS2 can be used for mass forced stop and start of batch jobs. See this blog for more details.
Batch job interception
For more information on batch job interception, please read this blog.
Ignoring large spool output of a batch job
In some cases a batch job will generate a large amount of spool output which is clogging the spool system, but is never used. In this case you can set up the job to put the output to output device NULL. For setup read this blog.
Archiving spool as print list
If you want to store the spool as print list in the archive, please read this blog.
If you want to quickly check which user ID’s are use inside batch job steps, it is best to quickly go to table TBTCP. Here you can quickly see across many batch jobs which step users are used.
Check batch job headers
Sometimes you need to report and check data on batch job header level. This data is stored in table TBTCO. Example: if you have performance issues with jobs and you have multiple application servers, you can use table TBTCO to check on which application server the troublesome jobs are running.
Solving inconsistencies in batch job tables
In rare cases there might be inconsistencies in the batch job tables TBTCO and TBTCS. In this case you need to run program RSBTCCNS to check and delete the inconsistencies.
You can mail the spool result of a batch job to one or more mail addresses. Also with newer SAP netweaver versions you can mail if job was successful or not. Read this blog in detail.
Debugging batch jobs
For debugging batch jobs, please read this dedicated blog.
Batch jobs full
If there are no batch processes free any more, you might see this screen in SM37:
When you face this situation check in SM66 if there are BTC processes available.
Standard batch jobs in S4HANA are planned via transaction SJOBREPO. For more details read the dedicated blog.
Batch job server groups
If you have a large production system with many application servers you can setup batch job server groups to have batch jobs run on a set of dedicated application servers. To set up batch job server groups start transaction SM61 and click the button Job server groups:
In quite some cases the basis team is asked: in which batch job variant is this company code XXXX used? Or we need to add another sales organization to all the batch jobs, can you provide us a list with jobs using sales organization YYYY? How to get this data is answered in this dedicated blog.
How to check health of batch job scheduler
Start transaction SM61. Now select the Time-Driven scheduler. Select the server and choose the health check tab. Now press the check button:
Batch job event triggers
Batch jobs can also be triggered using events. Read more in this blog on batch job event triggers.
Mass job changes
Mass job changes can be done via program BTC_MASS_JOB_CHANGE. Changes with mass changes options variants, step and job user, execution target.
Batch job monitoring
Batch job monitoring can be done with SAP Focused Run. Read more in this blog.
Transaction codes
Transaction codes that can help:
SM37: generic batch job transaction
SM37DSIP: generic batch job transaction, display only
Second program you need to run on production. So you first need to transport all the notes to production. The notes are large and can take up to 20 minutes import time. So take a quiet time at the productive system to import them.
Start program RC_COLLECT_ANALYSIS_DATA in your productive system:
Select your target version and press execute. This will launch multiple batch jobs. Wait for all the jobs to finish and start program again to download the ZIP file to your PC.
The newer version of S4HANA readiness report has option analyze the financial data quality. This is important, since a lot of time and effort during the conversion is spend on the financial data migration.
Hit the start New Analysis button and upload the file from the RC_COLLECT_ANALYSIS_DATA program. Wait for the SAP site to finish the analysis. It will be visible on the site as Recent Analysis.
Don’t forget to push the update analysis button to upload the second file with the custom code analysis from program SYCM_DOWNLOAD_REPOSITORY_INFO.
SAT ABAP runtime analysis is a great tool for analyzing performance issues. This tool can also be used to analyze ABAP web dynpro and FIORI applications. For more background on SAT tool itself, read this blog.
Questions that will be answered in this blog are:
How to run SAT tool for ABAP web applications?
Running SAT tool for ABAP web applications
Start transaction SAT, and press the button Schedule in the block For User/Service.
You now reach the measurement overview screen:
Now select the Schedule Measurement button. In the next screen fill out the user to trace and very important: switch the object type to URL and process type to HTTP:
Do not press the schedule measurement button yet. First go to the ABAP web application to the part you want to measure for performance. Afterwards delete the browser, since it will keep on sending data.
No you have to go back to the SAT start screen and click on the tab Evaluate:
Double click on your measurement line. The system will now read the measurement log files and process them. This can take some time. The end result screen looks again like a normal SAT result screen:
The SAT ABAP runtime analysis tool can be used to identify performance problems in ABAP programs.
Questions that will be answered in this blog are:
How to run the SAT tool?
How to read the results of the SAT tool?
Starting the SAT tool
The SAT ABAP runtime analysis tool can be started with transaction SAT:
Top left there is a Tips & Tricks button. This will bring you the to the following tool:
Here you can compare the optimal and not optimal way of coding. By hitting measure runtime button you can actually compare in real time the difference between the 2 methods.
The performance issue program
To test the tool, we first write a simple test program:
REPORT zperftest2.
DATA: zlt_vbak TYPE TABLE OF vbak.
DATA: zls_vbak TYPE vbak.
DATA: zlt_vbap TYPE TABLE OF vbap.
DATA: zls_vbap TYPE vbap.
DATA: zls_vbap2 TYPE vbap.
SELECT * FROM vbak INTO TABLE zlt_vbak UP TO 100 ROWS.
LOOP AT zlt_vbak INTO zls_vbak.
SELECT * FROM vbap INTO zls_vbap.
DO 10000 TIMES.
zls_vbap2 = zls_vbap.
ENDDO.
ENDSELECT.
ENDLOOP.
Now we start the SAT tool, enter the program name. Make sure the tick box evaluate immediately is on and press Execute.
Now the measurement will start.
Result of the trace tool
The result of the trace tool is as follows:
On the left side you see the split in where the program spends it time. Here you can see that most of the time is spend on internal processing and not on SQL statements. SQL statement can be analyzed from the SAT tool or from the ST05 SQL trace tool.
By double clicking on the the internal access the right hand side of the screen is filled. Here you can see in which code blocks the most net and gross time is spent. It does not always point you to the exact statements that are not ok, but it can point you to the program that is causing the biggest delay.
In our case the DO 10000 TIMES loop is the performance killer. With only SQL tracing this cannot be found.
Runtime analysis for web applications
For runtime analysis for web applications, read this dedicated blog.
Relevant OSS notes
Check and if needed you can apply these OSS notes to solve bugs in the SAT tool:
Performance issue solving is an important job for ABAP and basis consultant. One of the helpful tools is ST05 SQL performance trace tool.
Questions that will be answered in this blog are:
How to switch on ST05 SQL trace?
How to read the results from the ST05 SQL trace?
How to use the ST05 SQL trace with multiple application servers?
The test program
First we create a small test program for our performance analysis.
REPORT zperftest.
DATA: zlt_vbak TYPE TABLE OF vbak.
DATA: zls_vbak TYPE vbak.
DATA: zlt_vbap TYPE TABLE OF vbap.
DATA: zls_vbap TYPE vbap.
SELECT * FROM vbak INTO TABLE zlt_vbak UP TO 1000 ROWS.
LOOP AT zlt_vbak INTO zls_vbak.
SELECT * FROM vbap INTO zls_vbap.
DO 10 TIMES.
CALL FUNCTION 'BAPI_MATERIAL_EXISTENCECHECK'
EXPORTING
material = zls_vbap-matnr+0(18).
ENDDO.
ENDSELECT.
ENDLOOP.
Off course this program is written very badly to be really slow and making performance issues.
ST05 SQL trace
Start the ST05 SQL trace tool with transaction code ST05.
You can activate the trace for all users with the button Activate trace. But this is usually not needed and will only give you a long trace to analyze.
The usual method is to activate the trace with a filter, mostly on user name to trace a specific user action:
If the authorization team already started on ST01 authorization trace, you cannot start the SQL trace. First stop the ST01 trace.
Now you can start the performance test program in a different session. Just wait until it is finished. It is important the user you are tracing is only doing this action and does not perform any other action, since these other actions are also recorded and will make the analysis harder to read.
When completed go back to the ST05 and click Deactivate trace.
Trace analysis based on duration
Now press the Display trace to see the results. If your trace is large you will get this screen:
This is quite common and click yes. If your result is really too big, change the 5000 on the first screen in to a good maximum, or try to record a smaller time frame.
Result now looks like this after sorting on the column duration:
You can see that the total SQL commands took 48 seconds in total. The longest SQL single command took 78 milliseconds (which is very fast, so long running SQL’s are not the issue in this example).
In your case you might find very large single SQL commands. By selecting the line and pressing the button Display ABAP call location, you can jump to the place in the coding that fired the SQL command.
Summarized trace analysis
Since we have many records and many small ones, we need to use the summary tools. In the trace analysis select menu path Trace and then Structure Identical statements. This will give the following surprisingly short overview:
In the third column you can see that there are 100 identical calls, and also a high number of redundant calls.
If you now click on the second line to the ABAP call position you jump to our badly performing program and see that the majority of the delay is coming from the do 100 times loop. The first line is coming from standard SAP in the BAPI call.
This happens in real live as well that not all calls will point you into the right program. You might have to check multiple lines to find the right program causing the issue.
Multiple application servers
Development systems normally have only 1 application server. Productive servers can have more. If you have to analyze a performance issue directly on production with ST05, you have to be aware that the performance trace is per application server. If needed, you can select in ST05 as specific server, or you can select all. The trace display is still per server.
How does a deadlock look like in DB01 Oracle lock monitor?
How can I solve the issue?
Deadlocks
A database deadlock can occur when 2 or more processes are trying to update each other and are also waiting for each other. This process will lead to infinite waiting.
DB01 Oracle lock monitor
With the use of transaction DB01 you can get overview of Oracle locks. A deadlock is observed when the status is on HOLD and more processes are in status WAIT.
Solving the deadlock can be done by the Oracle DBA on database level.
ABAP programs (standard SAP and custom) use the shared memory objects technology.
Questions that will be answered in this blog are:
How to set the shared objects memory size?
How to monitor usage of shared objects memory?
Monitoring shared objects
Shared objects can be monitored with transaction SHMM:
In the start screen you can see the objects now loaded and how much memory they consume. In the start screen you can also delete and invalidate the shared memory objects.
The deletion can lead to short dumps. Only use this in emergency cases and when you are really sure
In the second tab you can see the total memory overview:
Setting shared memory objects memory size
In RZ11 with system parameter abap/shared_objects_size_MB you can set the shared object size in MB. A restart of the system is required to activate any change.
Older documentation says 500 or 1000 MB is sufficient. In newer systems a value of 3000 to 5000 in not uncommon.
SAP solution manager licenses have been renewed last few years by SAP.
Questions that will be answered in this blog are:
Do I need a user license for solution manager users?
If I run solution manager on HANA, do I need to pay HANA database licenses?
How can I get Focused Build and Focused Insights?
What about Focused Run licenses?
User licenses for SAP solution manager
Since 1.1.2018 the requirement of having named users was dropped by SAP.
HANA database licenses
If you want to run HANA database below SAP solution manager as database, you need to procure the infrastructure. The HANA database rights are included in SAP solution manager. This is the only exception SAP has. For all other use case you need to pay for HANA as database as well.
Using SAP solution manager for non-SAP components
You can use SAP solution manager to manage non-SAP components as well. Especially the ITSM service desk component can be used for this. When you use this function for non-SAP components, you will need SAP enterprise support rights for SAP solution manager in stead of the SAP standard support.
Focused Insights and Focused Build
SAP offers Focused Insights and Focused Build as extra options on top of SAP solution manager. Both are installed as add-on. Focused Insights brings extra dashboard building capabilities. With Focused Build you can get an extra grip on your solution build process.
If you want to try out these solutions, you can use the available free SAP demo system. Read more about this in the following blog.
SAP Focused run now also covers the functionality of Focused Insights, but in a far superior and more performing way. Read more in this blog.
Focused Run
Focused Run is separate solution with separate license to optimize the running of large SAP landscapes. Focused run does NOT run on SAP solution manager. It runs on a separate environment and only runs on SAP HANA. You cannot combine a Focused Run and SAP solution manager on one single installation. More information on Focused Run can be found on the SAP site. And on the specialized SAP Focused Run Guru site.
For licenses of Focused Run, read this dedicated blog.
Despite the fact that Focused Run is a paid solution, it offers by far the most sophisticated and added value product.
