In this case a dialogue with Dom Brooks on my previous blog on how to simulate a stored outline/SQL baseline with FORCE_MATCH using SQL profiles has provided me with a simplified way of approaching the creation of said profiles.

The observation is that there is a column OTHER_XML that appears in a number of tables and views from which you can directly copy the outline hints necessary to fix the execution plan of a statement – and use FORCE_MATCH to boot.

It seems that we are not the first to experiment with this technique. See for example this blog by Kerry Osborne published a while ago but I think these posts have slightly different emphasis.

The OTHER_XML column appears in PLAN_TABLE/SYS.PLAN_TABLE$, V$SQL_PLAN, and DBA_HIST_SQL_PLAN. The outline hints from these views are often displayed using the OUTLINE or ADVANCED formatting options to DBMS_XPLAN.DISPLAY, DBMS_XPLAN.DISPLAY_CURSOR, or DBMS_XPLAN.DISPLAY_AWR table functions respectively.

There is also an OTHER_XML column in DBA_SQLTUNE_PLANS that provides an alternative way to accept a SQL profile. Let me demonstrate. Let us setup some tables first:

-- 
-- Setup tables for test and gather stats 
-- 
DROP TABLE t1; 
DROP TABLE t2; 
DROP TABLE t3; 

BEGIN 
   FOR r IN (SELECT name FROM dba_sql_profiles) 
   LOOP 
      DBMS_SQLTUNE.drop_sql_profile (r.name); 
   END LOOP; 
END; 
/ 

CREATE TABLE t1 
AS 
       SELECT ROWNUM * 2 c1 
         FROM DUAL 
   CONNECT BY LEVEL <= 100000; 

CREATE TABLE t2 
AS 
       SELECT ROWNUM * 2 - 1 c1 
         FROM DUAL 
   CONNECT BY LEVEL <= 50000 
   UNION ALL 
   SELECT 2 FROM DUAL; 

CREATE TABLE t3 
AS 
       SELECT ROWNUM * 2 c1 
         FROM DUAL 
   CONNECT BY LEVEL <= 200000; 

EXEC dbms_stats.gather_table_stats(user,'T1'); 
EXEC dbms_stats.gather_table_stats(user,'T2'); 
EXEC dbms_stats.gather_table_stats(user,'T3'); 

CREATE INDEX t3_i1 
   ON t3 (c1); 

These tables are setup in way to deliberately confuse the CBO. Now, let us run the SQL Tuning advisor.

 
BEGIN 
   DBMS_SQLTUNE.drop_sql_profile (name => 'Tony''s SQLTUNE profile'); 
EXCEPTION 
   WHEN OTHERS 
   THEN 
      NULL; 
END; 
/ 

VARIABLE task_name VARCHAR2(20) 

BEGIN 
   :task_name := 
      DBMS_SQLTUNE.create_tuning_task ( 
         sql_text => ' 
SELECT * 
        FROM t1, t2, t3 
       WHERE t1.c1 = t2.c1 AND t1.c1 = t3.c1 
 '); 
   DBMS_SQLTUNE.execute_tuning_task (:task_name); 
END; 
/ 

SET LONG 100000 LONGC 100000 PAGES 10000 

SELECT DBMS_SQLTUNE.report_tuning_task (:task_name) FROM DUAL 
/ 

-- 
-- This is the standard way to accept a profile (with FORCE_MATCHING) 
-- 

--BEGIN 
--   DBMS_SQLTUNE.accept_sql_profile (task_name => :task_name, REPLACE => TRUE, name => 'Tony''s SQLTUNE profile', force_match => TRUE);

--END; 
--/ 

--EXPLAIN PLAN 
--   FOR 
--      SELECT * 
--        FROM t1, t2, t3 
--       WHERE t1.c1 = t2.c1 AND t1.c1 = t3.c1; 

--SELECT * FROM TABLE (DBMS_XPLAN.display); 

 

DECLARE 
   v_sqlprof_xml   CLOB; 
BEGIN 
   SELECT REGEXP_SUBSTR (other_xml, '<outline_data>.*</outline_data>') 
     INTO v_sqlprof_xml 
     FROM dba_sqltune_plans JOIN dba_advisor_tasks USING (task_id) 
    WHERE     attribute = 'Using SQL profile' 
          AND other_xml IS NOT NULL 
          AND task_name = :task_name; 

   DBMS_SQLTUNE.import_sql_profile (name => 'Tony''s SQLTUNE profile' 
                                    , description => 'Example profile from Tony Hasler''s blog' 
                                    , category => 'DEFAULT' 
                                    , sql_text => q'[ 
SELECT * 
        FROM t1, t2, t3 
       WHERE t1.c1 = t2.c1 AND t1.c1 = t3.c1 
]' 
                                    , REPLACE => TRUE, 
                                    force_match => TRUE, 
                                    profile_xml => v_sqlprof_xml); 
END; 
/ 

EXPLAIN PLAN 
   FOR 
      SELECT * 
        FROM t1, t2, t3 
       WHERE t1.c1 = t2.c1 AND t1.c1 = t3.c1; 

SELECT * FROM TABLE (DBMS_XPLAN.display); 

Stored outlines were introduced in Oracle 9i as a way of helping stabilise execution plans.  In 11gR1 these are deprecated in favour of enterprise-edition-only SQL Baselines (sorry standard edition users )-:) but as of 11gR2 neither facility has the capability built into SQL profiles for matching a plan to any statement that matches text when literal values and whitespace are ignored.  I heard this raised as an enhancement request at last week’s Uk Oracle User Group conference.

By a strange coincidence, I was faced with this precise problem this week.  The issue arose with a large application that was very time consuming and expensive to regression test.  It involved a dynamically generated SQL statement against partitioned tables that avoided the use of bind variables for literals, including a couple of dates.  Apart from the literal values, the query was identical from call to call.

When historic data was queried, the execution plan was fine.  However, when the current day was queried the plan was often very poor.  Neither gathering statistics during the day nor copying partition statistics from one day to the next helped.

My long term plan is to identify why the statistics copying isn’t working well and either to customise the copy process or to abandon the use of partition level statistics altogether.  But that is root cause analysis.  What about a recovery plan for my client?

The funny thing is that SQL baselines, stored outlines, and SQL profiles all store hints.  Normally, the types of hints that are stored in a SQL profile are very different from those stored in a SQL baseline or in a stored outline but they don’t have to be.

Let me demonstrate with a two table join for Christmas Day:

CREATE TABLE t1 (d1, c1 DEFAULT 'x')
PARTITION BY RANGE
   (d1)
   (
      PARTITION p1 VALUES LESS THAN (DATE '2012-01-01'),
      PARTITION pdefault VALUES LESS THAN (maxvalue))
AS
       SELECT DATE '2011-12-25', RPAD ('x', 2000)
         FROM DUAL
   CONNECT BY LEVEL <= 10000; 

CREATE TABLE t2
AS
       SELECT DATE '2011-12-25' d1
         FROM DUAL
   CONNECT BY LEVEL <= 100; 

EXEC dbms_stats.gather_table_stats(ownname=>user,tabname=>'T1',no_invalidate=>false);
EXEC dbms_stats.gather_table_stats(ownname=>user,tabname=>'T2',no_invalidate=>false); 

EXPLAIN PLAN
   FOR
      SELECT *
        FROM t1 NATURAL JOIN t2
       WHERE d1 = DATE '2011-12-25'; 

SET PAGES 0 LINES 132 

SELECT * FROM TABLE (DBMS_XPLAN.display (NULL, NULL, 'outline'));

This is the output I get on my 11.2.0.1 test database:

Plan hash value: 937381588             

----------------------------------------------------------------------------------------------------
| Id  | Operation                | Name    | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
----------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT         |         |  1000K|  1923M|   932   (1)| 00:00:12 |       |       |
|   1 |  HASH JOIN               |         |  1000K|  1923M|   932   (1)| 00:00:12 |       |       |
|   2 |   PART JOIN FILTER CREATE| :BF0000 |   100 |   800 |     3   (0)| 00:00:01 |       |       |
|   3 |    TABLE ACCESS FULL     | T2      |   100 |   800 |     3   (0)| 00:00:01 |       |       |
|   4 |   PARTITION RANGE SINGLE |         | 10000 |    19M|   923   (1)| 00:00:12 |KEY(AP)|KEY(AP)|
|   5 |    TABLE ACCESS FULL     | T1      | 10000 |    19M|   923   (1)| 00:00:12 |     1 |     1 |
---------------------------------------------------------------------------------------------------- 

Outline Data
-------------                          

  /*+
      BEGIN_OUTLINE_DATA
      USE_HASH(@"SEL$58A6D7F6" "T1"@"SEL$1")
      LEADING(@"SEL$58A6D7F6" "T2"@"SEL$1" "T1"@"SEL$1")
      FULL(@"SEL$58A6D7F6" "T1"@"SEL$1")
      FULL(@"SEL$58A6D7F6" "T2"@"SEL$1")
      OUTLINE(@"SEL$1")
      OUTLINE(@"SEL$2")
      MERGE(@"SEL$1")
      OUTLINE_LEAF(@"SEL$58A6D7F6")
      ALL_ROWS
      DB_VERSION('11.2.0.1')
      OPTIMIZER_FEATURES_ENABLE('11.2.0.1')
      IGNORE_OPTIM_EMBEDDED_HINTS
      END_OUTLINE_DATA
  */

Now what if I re-gather statistics and issue the same statement for New Years day?

EXEC dbms_stats.gather_table_stats(ownname=>user,tabname=>'T1',no_invalidate=>false);
EXEC dbms_stats.gather_table_stats(ownname=>user,tabname=>'T2',no_invalidate=>false);
EXPLAIN PLAN
   FOR
      SELECT *
        FROM t1 NATURAL JOIN t2
       WHERE d1 = DATE '2012-01-01'; 

SELECT * FROM TABLE (DBMS_XPLAN.display (NULL, NULL, 'outline -predicate'));

This is the plan for that date:

Plan hash value: 2331183230
------------------------------------------------------------------------------------------------
| Id  | Operation               | Name | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT        |      |     1 |  1019 |     6  (17)| 00:00:01 |       |       |
|   1 |  HASH JOIN              |      |     1 |  1019 |     6  (17)| 00:00:01 |       |       |
|   2 |   PARTITION RANGE SINGLE|      |     1 |  1011 |     2   (0)| 00:00:01 |     2 |     2 |
|   3 |    TABLE ACCESS FULL    | T1   |     1 |  1011 |     2   (0)| 00:00:01 |     2 |     2 |
|   4 |   TABLE ACCESS FULL     | T2   |     1 |     8 |     3   (0)| 00:00:01 |       |       |
------------------------------------------------------------------------------------------------
Outline Data
-------------
  /*+
      BEGIN_OUTLINE_DATA
      USE_HASH(@"SEL$58A6D7F6" "T2"@"SEL$1")
      LEADING(@"SEL$58A6D7F6" "T1"@"SEL$1" "T2"@"SEL$1")
      FULL(@"SEL$58A6D7F6" "T2"@"SEL$1")
      FULL(@"SEL$58A6D7F6" "T1"@"SEL$1")
      OUTLINE(@"SEL$1")
      OUTLINE(@"SEL$2")
      MERGE(@"SEL$1")
      OUTLINE_LEAF(@"SEL$58A6D7F6")
      ALL_ROWS
      DB_VERSION('11.2.0.1')
      OPTIMIZER_FEATURES_ENABLE('11.2.0.1')
      IGNORE_OPTIM_EMBEDDED_HINTS
      END_OUTLINE_DATA
  */

Now what if I know that the first plan is desirable and the second one isn’t? Well, I can load the outline hints associated with the good plan into a SQL profile:

BEGIN
   DBMS_SQLTUNE.import_sql_profile (
      name          => 'BLOG profile',
      description   => 'Example profile from Tony Hasler''s blog',
      category      => 'DEFAULT',
      sql_text      => q'[
 SELECT *
        FROM t1 NATURAL JOIN t2
       WHERE d1 = DATE '2011-12-25'
   ]',
      REPLACE       => TRUE,
      force_match   => TRUE,
      profile       => sqlprof_attr('BEGIN_OUTLINE_DATA',
'USE_HASH(@"SEL$58A6D7F6" "T1"@"SEL$1")',
'LEADING(@"SEL$58A6D7F6" "T2"@"SEL$1" "T1"@"SEL$1")',
'FULL(@"SEL$58A6D7F6" "T1"@"SEL$1")',
'FULL(@"SEL$58A6D7F6" "T2"@"SEL$1")',
'OUTLINE(@"SEL$1")',
'OUTLINE(@"SEL$2")',
'MERGE(@"SEL$1")',
'OUTLINE_LEAF(@"SEL$58A6D7F6")',
'ALL_ROWS',
'DB_VERSION(''11.2.0.1'')',
'OPTIMIZER_FEATURES_ENABLE(''11.2.0.1'')',
'IGNORE_OPTIM_EMBEDDED_HINTS',
'END_OUTLINE_DATA'));
END;

Careful editing is required: each hint has to be an element in the SQLPROF_ATTR array so be careful with multi-line hints. If you have a very long hint you may have to use the overloaded variant of this undocumented routine that allows you to specify XML rather than an array (profile_xml rather than profile). In my case, I was able to simply replace a long list of columns in an index hint with an index name. Also note the need to deal with sinqle quotes inside the strings (as in OPTIMIZER_FEATURES_ENABLE).

Note the important FORCE_MATCH parameter.

With this profile in place, we get the same plan for both Christmas Day and New Years Day.

You may wonder what happens if the SQL text already contains hints? Well, you have to include these embedded hints in the SQL text when you create the profile but the BEGIN_OUTLINE_DATA, IGNORE_EMBEDDED_HINTS, and END_OUTLINE_DATA hints in the SQL profile combine to allow the embedded hints to be overridden.

First from section 3.4 on isolation requirements:

P1 (“Dirty Read”): Database transaction T1 modifies a data element. Database transaction T2 then reads that data element before T1 performs a COMMIT. If T1 were to perform a ROLLBACK, T2 will have read a value that was never committed and that may thus be considered to have never existed.

A little ater on it says that dirty reads are not allowed, as you might expect.

skipping ahead a bit…

Now, in section 3.5.2 we have the committed property definition:

A transaction is considered committed when the transaction manager component of the system has either written the log or written the data for the committed updates associated with the transaction to a durable medium.

Therefore, Oracle violates the isolation property because data not yet committed has been made visible to other transactions and “Dirty Reads” are not prevented.  Specifically, the read-only order status transaction may see changes made by the other read-write transactions listed above.

The purpose of this blog entry is to explain why this is such a critical issue.  There are three main areas I want to touch on (in increasing order of importance):

- Giving humans invalid data
- Giving computers invalid data
- Legal and reputational issues

Let us begin with the human case.  You call your broker late at night with instructions:  “I hear that Oracle is not ACID.  Please sell my 10 million pounds worth of Oracle stock!”.  Your broker says, “OK, I have submitted your instructions for processing.  It might take a few minutes”.  You say…”I’ll hold.  I am going on vacation tomorrow and want to make sure that this is done before I leave”.  Your broker replies: “Let me query the system to check……oh yes your instructions are definitely in our system.  Your transaction to sell Oracle at best will be actioned tomorrow morning”.  You feel relieved.  However, the system crashed after the broker was sent confirmation that your transaction was processed but before the transaction was on disk.  Your request to sell Oracle stock is lost and you return from vacation to find that no record exists of your request.

This sort of scenario has probably happened a few times as a result of Oracle’s lack of ACID.  However, nobody would have known why.  On the other hand, it will be a very rare event.  Even an entire call centre will only make at most a few queries per second and most of these will be for data that was committed long ago.  It will be a rare event that sees queried data lost on a crash.

If this sort of situation has ever happened to you it is highly unlikely that Oracle’s lack of ACID was the cause.  Bugs in application code, media recovery and human error amongst other causes would account for the vast majority of such mishaps.

However, let us turn our attention to computer to computer communication.  Consider the following situation.  You process some kind of incoming data and place the data in a staging table.  Let us assume that the table is partitioned by hash into 256 segments. You have 256 producer threads that:

- read data from their own partition
- enrich it in a few ways (including adding a primary key generated from a sequence)
- send the enriched data to an AQ queue
- delete the data from the staging table
- commit.
- loop

Meanwhile, you have another group of sessions that read messages from the AQ queue and send data to a remote system that you have no control over.  Your consumer threads:

- Destructively read a message from the AQ queue
- Insert the data into the remote system (using the remote systems transaction system)
- Commit the remote transaction
- Commit the local transaction.
- loop

You are a smart guy or gal and realise that there is an issue with your code:  the system may crash after the remote transaction has committed and before the local one does.  This means that after a recovery the message will be back in AQ and will be sent again.  However, there is a unique primary key in the AQ message and the remote system is built in such a way as to detect these “possible duplicates” and discards the message when it is resent.

Wonderful, but because of the failure of ACID it is possible that not only the uncommitted consumers transaction is lost on a crash but also the “committed” producers transaction!  Now on recovery, the message would not be in the AQ queue but back in the staging table.  Now when the message is resent it will have a brand new primary key and the remote system will not detect the duplicate.

There are lots of systems where the log writer is the bottleneck, perhaps because of replication to a DR site or for other reasons.  In these sorts of heavily loaded systems it is virtually guaranteed that at the time of a crash there will be some transactions that are sitting the log buffer waiting to be written out to disk.  When automated tasks are processing transactions immediately after that have been committed recovery will create a problem nearly all the time.

What is so bad about this situation is that there is no easy way, short of a complete redesign, to fix the application to workaround this problem as Oracle provides no way for a session (other than the committing one) to wait for the data it is seeing to be on disk.  Sure, the consumer threads could wait a few seconds before sending the data to the remote host but this would kill performance.

It is the potentially high rate that consumer threads can read “undurable data” that makes the probability of an error high in these situations.

The final, and most important, area is the potential legal and reputation issues that this topic raises.  It will be very difficult for Oracle to claim that this is a documentation error and amend same to say that Oracle is NOT ACID.  On the other hand, it is very difficult to see how they can treat this as a simple “bug” as it is a fundamental part of the design of the product and any “bug fix” would seem to have massive performance implications.

When I first wrote this blog the intention was to discuss the few circumstances when a COMMIT from a PL/SQL might impact ACID properties and to explain why 99% of the time it didn’t.  At the time I wrote this blog the assumption was that ACID properties could confidently be assigend to standard COMMIT statements executed from SQL (e.g. SQL*PLUS, JDBC, etc.).  As you will see in the comments to this blog this is far from the case.  I have annoted the false assumption in bold.

The first thing to get your head round is that PL/SQL language statements are different from SQL statements. When you issue a COMMIT statement from SQL*PLUS, from Java, C, or almost all other languages you need to refer to the “SQL Language Reference” manual for syntax and semantics. When you issue a COMMIT statement (or one of the other seven SQL statements with direct PL/SQL equivalents) from PL/SQL, however, you need to look in “PL/SQL Language Reference” manual for syntax and semantics. Sure, they do very similar things but there are subtle differences.

In the 11.2 SQL Language Reference manual, you will see this statement:

If you omit [the write clause], then the behavior of the commit operation is controlled by the COMMIT_LOGGING and COMMIT_WAIT initialization parameters, if they have been set.

The COMMIT_LOGGING and COMMIT_WAIT parameters replace the COMMIT_WRITE parameter used in previous releases. Unfortunately, the documentation for COMMIT_WRITE is at best incomplete and at worst wrong. To see what the default semantics are we need to look at the documentation for the deprecated COMMIT_WRITE parameter. Here we can see:

If this parameter is not explicitly specified, then database commit behaviour defaults to writing commit records to disk before control is returned to the client.

We also see this line:

Be aware that the NOWAIT option can cause a failure that occurs after the database receives the commit message, but before the redo log records are written. This can falsely indicate to a transaction that its changes are persistent. Also, it can violate the durability of ACID (Atomicity, Consistency, Isolation, Durability) transactions if the database shuts down unexpectedly.

This is poor English but obviously sounds scary. Most programmers would and should avoid using NOWAIT in their programs or overriding the default session or system initialisation settings. However, let us look at the documentation in the PL/SQL reference manual:

The default PL/SQL commit behaviour for nondistributed transactions is BATCH NOWAIT if the COMMIT_LOGGING and COMMIT_WAIT database initialization parameters have not been set.

Whoa! Suddenly the behaviour that you were warned might cause a violation of the durability of durability (that is what it says isn’t it?) is now the default behaviour! Why is something that is so dangerous from SQL suddenly safe from PL/SQL and why?

Well, as we shall see it isn’t always safe. But to understand what is safe and what is not safe let me try to expand on the scary documentation from the SQL language reference manual. Consider this piece of pseudo-code from some non-PL/SQL program:

   do some DML;
   commit;
   put_on_screen('Transaction complete') ;

Imagine that the commit returned immediately without waiting for the data to be recorded on disk. You might tell the user that the transaction committed and then the database might crash causing all the DML to be lost. You have misled the user. What if, however, the ‘user’ was another computer system or database?

    get a message from a remote system;
    do some DML;
    commit;
    send confirmation of receipt of message to remote system;

Well, a human might (or might not) realise that there was a crash around the time his message was displayed saying that his transaction completed but a remote computer system will not be so paranoid. It is highly unlikely that such a remote application would retransmit a message if it receives an explicit positive acknowledgement to the first transmission.

So this is why the default behaviour for SQL is to wait until the transaction details are recorded in the redo log on disk; it is to ensure that if the application relays the fact that the transaction is committed to a third party (human or computer) that the transaction isn’t later discovered to be ‘uncommitted’ so to speak. This is the “durability” part of ACID.

Now that we have that sorted out let us turn our attention to PL/SQL. First of all, the documentation here is “incomplete”. Let me try and expand:

• For all but the last commit statement executed before PL/SQL returns control to the caller the default PL/SQL commit behaviour for nondistributed transactions is BATCH NOWAIT if the COMMIT_LOGGING and COMMIT_WAIT database initialization parameters have not been set.
• The default behaviour for the last commit statement executed before PL/SQL returns control to the caller is to trigger a wait at the point control is returned to the caller.
• Sometimes you will experience a ‘log file sync’ wait event even if you explicitly or implicitly specify NOWAIT.

Ok, so the semantics are not only apparently dangerous but also complicated. Why?

Well let us take an example PL/SQL block and add some measurements of ‘log file sync’wait events.  Let us begin by creating a test table:

CREATE TABLE T1 (dummy varchar2(100)) ;

Now our test:

 SELECT total_waits
  FROM v$session_event
 WHERE sid = USERENV ('SID') AND event = 'log file sync';

BEGIN
   INSERT INTO t1 (dummy)
        VALUES ('Transaction 1');

   COMMIT;

   INSERT INTO t1 (dummy)
        VALUES ('Transaction 2');

   COMMIT;
END;

SELECT total_waits
  FROM v$session_event
 WHERE sid = USERENV ('SID') AND event = 'log file sync';

We see only one ‘log file sync’. This is the one triggered at the end of the block by the second commit. The first commit triggered no wait. However, no risk of a consistency problem arises. This is because:

• Locking will prevent either inserted row being visible to outside sessions until the redo is actually on disk and

The statement above is how databases in general are supposed to work.  Oracle does not work this way, however: commits from PL/SQL or SQL are visible to other sessions before they are actually on disk.  See the comments below for more discussion.

• The second transaction will have a higher SCN than the first and in the event of a crash Oracle will ensure that if the first transaction needs to be rolled back the second will be also.

This can be a bit tricky to get your head round but it is a legitimate optimisation that I believe (historians are welcome to correct me) was first introduced in IMS Fastpath many decades ago and has been copied by Oracle and other database systems subsequently.

When I have explained this to other DBAs and programmers over the years I have known when the point has been understood when I get the question:

“Yes, but what if I communicate the commit outside of the PL/SQL block by some means other than updating the database”?

This question indicates a grasp of the key concept. Unfortunately, I have always answered this question incorrectly! Let us get one easy case out of the way first:

BEGIN
   INSERT INTO t1 (dummy)
        VALUES ('Transaction 1');

   COMMIT;
   DBMS_OUTPUT.put_line ('Transaction committed');
END;

This is actually safe because the results of DBMS_OUTPUT.PUT_LINE are invisible until after the PL/SQL block returns. So if the supposedly committed transaction is subsequently rolled back nobody will ever have known that the commit statement returned to the PL/SQL block in the first place.

But what about this:

SELECT total_waits
  FROM v$session_event
 WHERE sid = USERENV ('SID') AND event = 'log file sync';

DECLARE
   filehandler   UTL_FILE.file_type;
BEGIN
   INSERT INTO t1 (dummy)
        VALUES ('Transaction 1');

   COMMIT;

   INSERT INTO t1@dblink (dummy)
        VALUES ('Transaction 2');

   COMMIT;
   filehandler :=
      UTL_FILE.fopen (
                      'DATA_PUMP_DIR',
                      'test_file.txt',
                      'W'
                     );
   UTL_FILE.putf (
                  filehandler,
                  'A transaction committed\n'
                 );
   UTL_FILE.fclose (filehandler);

   INSERT INTO t1 (dummy)
        VALUES ('Transaction 3');

   COMMIT;
END;

SELECT total_waits
  FROM v$session_event
 WHERE sid = USERENV ('SID') AND event = 'log file sync';

You might have to run this a couple of times as there is sometimes an ‘extra’ ‘log file sync’ wait event but most of the time there will only be one wait for the entire PL/SQL block. The use of a database link doesn’t mean there is a distributed transaction because there are no changes to the local database in the transaction. If the local database crashes neither the update to the text file nor the update to the remote database will be “rolled back” so an inconsistency can arise after all.

Until I started to prepare for writing this blog I had always assumed that whenever a remote database was updated or a package such as UTL_FILE or UTL_HTTP was called then a ‘log file sync’ would occur at that point. This is such a fundamental requirement of database consistency, it never occurred to me to check. Not only that, but the PL/SQL engine knows enough to do this easily. However, even in 11.2 this is not the case. You can protect against these inconsistencies by changing your PL/SQL COMMIT statements (the last being optional) to ‘COMMIT WORK WRITE WAIT’ statements.

So when is it safe to use COMMIT WORK WRITE NOWAIT (either explicitly or not)?

• If the only thing you do between ‘commit number one’ and ’commit number two’ is issue queries and/or updates to the local database it is safe for ‘commit number one’ to use NOWAIT. This is true both for PL/SQL and SQL.
• If you commit a transaction and then your program terminates completely then it is safe to use NOWAIT. This is true for both SQL and PL/SQL.
• If you commit a transaction and then ‘communicate with the outside world’ then you should wait (explicitly in PL/SQL) if you care about consistency.

http://kevinclosson.wordpress.com/2007/07/21/manly-men-only-use-solid-state-disk-for-redo-logging-lgwr-io-is-simple-but-not-lgwr-processing/.

However, it seems that everybody has missed a fundamental complication of the changes in commit processing introduced in 10g.  The point has probably been missed because it is normally very hard to demonstrate.

Oracle 9i and below Oracle had the concept of a ‘log buffer’.  You would be in good company if you thought that later versions had a log buffer as well.  After all, there is an initialisation parameter LOG_BUFFER that specifies how big it is.

In reality, Oracle now uses a combination of ‘private redo strands’ and ‘public redo strands’.

Private redo strands do not exist in instances within a RAC cluster.  In a non-RAC environment each transaction gets its own redo strand (subject to availability).  These strands are used in combination with in-memory undo (IMU) to reduce the amount of data that is logged but more importantly to reduce latching conflicts; data is normally copied from a private redo strand only at commit time.  Data will also be copied out if the private redo strand fills up before a commit is issued.  These private redo strands are allocated from shared pool.

RAC or no RAC there is also a minimum of two public redo strands as well.  In larger systems there may be more (number of CPUs divided by 16).  These public redo strands also contribute to latch reduction.  The number of these strands can be changed by use of the hidden parameter “_log_parallelism_max”.

In relatively quiet systems only one public redo strand may be in use but even with as few as two concurrent sessions busy generating redo you can see the second redo strand being used.  When that happens, LGWR writes out the data in the public redo strands sequentially!

How do I know this?

Well, I was performing a rather artificial insert-only performance test on one instance of an 11.2 RAC database on a 64-CPU system with a suspected faulty I/O subsystem (don’t ask).  LGWR was spending 90% of its time in ‘log file parallel write’ (LFPW) and about 10% of the time on the CPU.  I had taken the advice in Kevin’s article above and “reniced” LGWR so I was fairly sure that LGWR rescheduling was not a big issue: throughput was limited by the I/O subsystem.

Nevertheless, ‘log file sync’ (LFS) wait events averaged 4 or 5 times the LFPW times despite the average CPU utilisation only being about 25%.  Theoretically, one would expect that the average LFS time would be somewhat more than 50% higher than LFPW in a system that is limited by LGWR I/O:  if LGWR has just started an I/O at the time that you commit then you have to wait for that I/O to finish as well as your own that follows.  If you are luckier LGWR may just finish an I/O as you commit and now you just have to wait for your own I/O. This gives an average LFS/LFPW ratio of 1.5.  Given the fact that you are still in LFS when LGWR is on the CPU doing its post processing I thought that LFS should be a little higher than 50% above LFPW but certainly not 300% or 400% higher.

Explanations revolving around ‘CPU storms’ didn’t seem plausible so I had to dig deeper.  Well, by ‘digging deeper’ I mean a) posting a question on OTN and asking for help from ‘knowledgeable friends’.  When that failed b) guessing.  I eventually guessed correctly when I set “_log_parallelism_max” to 1 (it was previously 4 on my 64 CPU system).

The faulty I/O subsystem had (has) a characteristic that throughput is several times higher when the I/O size is larger.  The reason for this is not yet known.  However, when the initialisation parameter change was made:

• The average size of the I/O increased several times
• The (faulty) I/O subsystem throughput increased several fold
• The LFS time came back down to just over 50% of the LFPW time.
• LGWR started waiting on the idle event ‘rdbms ipc message’; the system was no longer limited by I/O.
• Latching increased thus proving that the long term solution was to fix the I/O subsystem not make unsupported changes to initialisation parameters!

I then set about trying to devise some kind of experiment that others can repeat to verify my conclusions.  The problem is that the only way to get more than one public redo strand in play is to have sufficient redo being generated that a redo allocation latch collision occurs.  You can easily generate this scenario.  You can also prove that the number of transactions being committed each write (redo size/commit size) is much less than the number of processes in LFS but I can’t think of a way to prove that this discrepancy is caused by I/O serialisation without a faulty I/O subsystem.  Sure you can run a trace and see that there are multiple LFPW events between idle ‘rdbms ipc message’ events but that is hardly conclusive.

Any ideas on how to devise an experiment would be warmly welcomed.  In the meantime I guess you will just have to take my word for it!

You are a programmer or a DBA and you are trying for the first time in your life to analyze an Oracle performance problem. What is the most important concept you need to understand to get started?

Well, there are a number of right answers to this question but rightly or wrongly the ‘The Oracle wait interface’ would be many people’s answer. A very basic approach would be to look at the highest wait event on the system and focus on that. So for example, if you see a lot of ‘db file sequential read’ wait events you might conclude that faster disks might be the answer. You may be a bit smarter and realise that these events are generally associated with indexed access and try and work out what indexes are being accessed and figure out if they (or the queries that use them) can be redesigned.

There are, of course, numerous books and web articles that explain the deficiencies of this type of approach but in many cases it seems to work. The success of this type of basic approach will be quite high in simple systems where performance has been neglected for a long time and there are one or two ‘quick wins’ that can have a significant benefit. In complex systems, or ones that have been properly managed over the years this sort of basic analysis is unlikely to be enough.

Cary Millsap wrote a book with Jeff Holt many years ago entitled ‘Optimizing Oracle Performance’ in which he explained the limitations of any system wide tuning approach. Even though the book was written many years ago, I would say that this is still mandatory reading for any Oracle performance specialist. The next most important book, in my opinion, is a more modern work by Christian Antognini http://antognini.ch/top/ but I digress.

Is ‘LOG FILE SYNC’ (LFS) relevant?

Arguably the single most misleading wait event in the Oracle database is ‘log file sync’ (LFS). Let me explain why. Suppose you have a single client process and it spends 1% of its time in LFS. Would you gain a lot by engaging in a tuning exercise that aimed exclusively at reducing this wait event? Well, obviously the upper limit on the gain you might make is 1% even if you eliminated these events in their entirety; the answer is, therefore, NO. You are unlikely to waste your time though because your Statspack or AWR report will probably not highlight the event in the first place.

On the other hand, suppose you have 10,000 sessions connecting to your database and 300 of these are active at any one time. Suppose that 1% of the active time of each session is spent in LFS would you gain much by tuning the event? The answer is the same as before: NO. However, you might be misled by Statspack or AWR because now the total wait time for the database for LFS will be 3 hours in any one hour! This is because at any one time we have 300 active sessions and on average three of them will be in LFS. That comes out to be 3 hours per hour waiting! So before you conclude that LFS is a major performance issue first check out what the average wait time per active session is. Of course, what you should really do is find out the percentage of wait time for your business critical processes but I digress again. Let us get back to the technicalities of LFS.

‘Log File Sync’ (LFS) versus ‘log file parallel write’ (LFPW)

When a transaction commits it will usually wait for its redo data to be recorded on disk before returning control to the application. That wait is LFS. There are three main exceptions to this:

• When the commit is issued within a PL/SQL block. In this case the wait will be deferred until the block returns. If the PL/SQL block contains multiple commit statements then only one wait will occur when the block returns.
• When the commit statement is actually COMMIT WORK NOWAIT. In this case, there is no LFS wait. See part 3 in this series for an explanation of why this is dangerous.
•  When the session parameter COMMIT_WAIT (COMMIT_WRITE in earlier releases) is set to NOWAIT. This parameter can be set at the instance level but this is dangerous practice and should be set at the session level only with care. See part 3 in this series for an explanation of why this is dangerous.

The waiting client process does not do the I/O itself. LGWR does. When LGWR does the I/O it waits on ‘log file parallel write’ (LFPW). So does this mean that the number of LFPW events should equal the number of LFS events? NO! This is because LGWR employs a tactic known as ‘group write’ and may write out the redo data for multiple transactions and multiple sessions in one go. Suppose you have 5 client processes each of which is waiting for its transaction to complete. These transactions will be waiting for LFS. LGWR will be waiting for LFPW. If the I/O takes 10ms then:

• The system will accumulate 50ms of time in LFS (10ms for each client process). 5 wait events will be recorded.
• LGWR will accumulate 10ms. One wait event will be recorded.

That is 6 wait events totalling 60ms clocked up for one 10ms I/O! You see the danger in oversimplifying?

In the simplified example above I have suggested that the average wait time for LFS should be about the same as the average wait time for LFPW as they are both associated with waiting for the redo data to be written to disk. In fact, this is itself a misleading simplification. Onto part 2.

Let us now address what happens when we try to use NULL for maximum/minimum values by modifying the script in part 1:

DROP TABLE t1;
CREATE TABLE t1 (a NUMBER, b DATE, c TIMESTAMP, d TIMESTAMP WITH LOCAL TIME ZONE,
 e RAW(32), f VARCHAR2(32),g NVARCHAR2(32),h CHAR(32),
i INTERVAL DAY TO SECOND,j INTERVAL YEAR TO MONTH, k ROWID,
l TIMESTAMP WITH TIME ZONE,m NCHAR(32));

DECLARE
   srec   DBMS_STATS.statrec;
BEGIN
   DBMS_STATS.set_table_stats (ownname            => USER,
                               tabname            => 'T1',
                               numrows            => 20000,
                               numblks            => 1000,
                               avgrlen            => 400,
                               no_invalidate      => FALSE);

   FOR r IN (SELECT *
               FROM user_tab_cols
              WHERE table_name = 'T1')
   LOOP
      srec.epc := 2;                       -- Two endpoints
      srec.bkvals := NULL;                  -- No histogram
      DBMS_STATS.delete_column_stats
                                  (ownname      => USER,
                                   tabname      => 'T1',
                                   colname      => r.column_name);
      DBMS_STATS.prepare_column_values
                                 (srec,
                                  DBMS_STATS.rawarray (NULL,

                                                       -- Minimum
                                                       NULL
                                                           -- Maximum
                                  ));
      -- srec.novals := dbms_stats.numarray(-9.9e125,9.9e125) ; -- Special fix
      DBMS_STATS.set_column_stats (ownname       => USER,
                                   tabname       => 'T1',
                                   partname      => NULL,
                                   colname       => r.column_name,
                                   distcnt       => NULL,
                                   density       => NULL,
                                   nullcnt       => 0,
                                   srec          => srec,
                                   avgclen       => 100);
   END LOOP;
END;
/

SET autotrace off
COLUMN lowval format a10
COLUMN hival format a10
SET null '(null)'

SELECT distcnt, density, h.lowval, h.hival, h.MINIMUM,
       h.maximum
  FROM SYS.obj$ o, SYS.hist_head$ h
 WHERE o.NAME = 'T1'
   AND o.owner# = USERENV ('SCHEMAID')
   AND o.obj# = h.obj#;

SET autotrace traceonly explain
SELECT *
  FROM t1
 WHERE a > -1;
SELECT *
  FROM t1
 WHERE a > 0;
SELECT *
  FROM t1
 WHERE a = 0;
SELECT *
  FROM t1
 WHERE a  0;
SELECT *
  FROM t1
 WHERE a = 1;
SELECT *
  FROM t1
 WHERE a  1;

The above script generates the following output:

   DISTCNT    DENSITY LOWVAL     HIVAL         MINIMUM    MAXIMUM
---------- ---------- ---------- ---------- ---------- ----------
         0          0 (null)     (null)              0          0
         0          0 (null)     (null)              0          0
         0          0 (null)     (null)              0          0
         0          0 (null)     (null)              0          0
         0          0 (null)     (null)              0          0
         0          0 (null)     (null)              0          0
         0          0 (null)     (null)              0          0
         0          0 (null)     (null)              0          0
         0          0 (null)     (null)              0          0
         0          0 (null)     (null)              0          0
         0          0 (null)     (null)              0          0
         0          0 (null)     (null)              0          0
         0          0 (null)     (null)              0          0

13 rows selected.

SQL>
SQL> SET autotrace traceonly explain
SQL> SELECT *
  2    FROM t1
  3   WHERE a > -1;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 20000 |  7812K|   223   (2)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 20000 |  7812K|   223   (2)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A">(-1))

SQL> SELECT *
  2    FROM t1
  3   WHERE a > 0;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |     1 |   400 |   222   (1)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   |     1 |   400 |   222   (1)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A">0)

SQL> SELECT *
  2    FROM t1
  3   WHERE a = 0;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |     1 |   400 |   222   (1)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   |     1 |   400 |   222   (1)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A"=0)

SQL> SELECT *
  2    FROM t1
  3   WHERE a  0;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 20000 |  7812K|   223   (2)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 20000 |  7812K|   223   (2)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A"0)

SQL> SELECT *
  2    FROM t1
  3   WHERE a = 1;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |     1 |   400 |   222   (1)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   |     1 |   400 |   222   (1)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A"=1)

SQL> SELECT *
  2    FROM t1
  3   WHERE a  1;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 19999 |  7812K|   223   (2)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 19999 |  7812K|   223   (2)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A"1)

SQL> select * from t1 where a  3000;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 19999 |  7812K|   223   (2)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 19999 |  7812K|   223   (2)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A"3000)

SQL> select * from t1 where a  -1;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 19999 |  7812K|   223   (2)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 19999 |  7812K|   223   (2)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A"(-1))

SQL> select * from t1 where a  0;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 20000 |  7812K|   223   (2)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 20000 |  7812K|   223   (2)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A"0)

SQL> select * from t1 where a < 1;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 20000 |  7812K|   223   (2)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 20000 |  7812K|   223   (2)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A" select * from t1 where a < 0;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |     1 |   400 |   222   (1)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   |     1 |   400 |   222   (1)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A" select * from t1 where a < -1;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |     1 |   400 |   222   (1)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   |     1 |   400 |   222   (1)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A" select * from t1 where a < 2;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 20000 |  7812K|   223   (2)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 20000 |  7812K|   223   (2)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A" select * from t1 where a <= 2;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 20000 |  7812K|   223   (2)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 20000 |  7812K|   223   (2)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A"<=2)

The first thing to note is that zeroes have been added for the DISTCNT, DENSITY, MINIMUM and MAXIMUM columns in HIST_HEAD$. This causes the selectivity estimate of queries involving a range predicate to be 100% if the range includes 0 and 0% (cardinality 1) when the range does not include 0. Equality and inequality predicates yield 0% and 100% respectively regardless of whether zero is specified as the value (although not the slight discrepancy in the inequality value when zero is not specified).

If you change the script to specify non-zero values for the density and a distcnt greater than 1 you will find that the behaviour changes as follows:

> -1 : 100%
> 0 : (1-density)*100%
= 0 : density*100%
<> 0 : (1-density)*100%
= 1 : density*100%
<> 1 : (1-density)*100%

The bottom line is that this mechanism is probably too complex to be of practical use and the technique of using very diffent values documented in part 1 of this blog will work much better.

During that presentation Wolfgang Breitling posed the question “What happens if you set the maximum and minimum values to NULL”?

I couldn’t answer him at the time and I promised that I would post an answer on my blog.

You will have to wait for the answer a little bit more I am afraid because I must first post some principles that I discovered whilst researching the answer.

If we dive down into the data dictionary we can find a a table called HIST_HEAD$. This table contains the minimum and maximum values for column statistics. What is interesting is that there are two sets of minimum and maximum values. These pairs of values are MINIMUM/MAXIMUM (type NUMBER) and LOWVAL/HIVAL (type RAW(32)). As far as I can see, only datatypes CHAR,NCHAR, VARCHAR2, and NVARCHAR2 use the LOWVAL/HIVAL pair in cardinality calculations. All other datatypes use the MINIMUM/MAXIMUM pair (including RAW!).

Unfortunately, when you look at views such as USER_TAB_COLS all you see is the LOWVAL/HIVAL pair from sys.HIST_HEAD$ reported as LOW_VALUE/HIGH_VALUE. The other pair is completely hidden.

“So what”? You may ask. These values are equivalent aren’t they?

Well some of the time, yes. But not always.

Here is how you might set MIN/MAX column statistics yourself:

DROP TABLE t1;
CREATE TABLE t1 (a NUMBER, b DATE, c TIMESTAMP, d TIMESTAMP WITH LOCAL TIME ZONE,
 e RAW(32), f VARCHAR2(32),g NVARCHAR2(32),h CHAR(32),
i INTERVAL DAY TO SECOND,j INTERVAL YEAR TO MONTH, k ROWID,
l TIMESTAMP WITH TIME ZONE,m nchar(32));

DECLARE
   srec   DBMS_STATS.statrec;
BEGIN
   DBMS_STATS.set_table_stats (ownname            => USER,
                               tabname            => 'T1',
                               numrows            => 20000,
                               numblks            => 1000,
                               avgrlen            => 400,
                               no_invalidate      => FALSE);

   FOR r IN (SELECT *
               FROM user_tab_cols
              WHERE table_name = 'T1')
   LOOP
      srec.epc := 2;                       -- Two endpoints
      srec.bkvals := NULL;                  -- No histogram
      DBMS_STATS.prepare_column_values
                                (srec,
                                 DBMS_STATS.rawarray (
                                 hextoraw('000000000000000000000000000000'),
                                                      -- Minimum
                                 hextoraw('ffffffffffffffffffffffffffffff')
                                                       -- Maximum
                                 ));
      -- srec.novals := dbms_stats.numarray(-9.9e125,9.9e125) ; -- Special fix
      DBMS_STATS.set_column_stats
                                 (ownname       => USER,
                                  tabname       => 'T1',
                                  partname      => NULL,
                                  colname       => r.column_name,
                                  distcnt       => 2,
                                  density       => 0.5,
                                  nullcnt       => 0,
                                  srec          => srec,
                                  avgclen       => 100);
   END LOOP;
END;
/
set autotrace off
column lowval format a30
column hival format a30
select h.lowval,h.hival,h.minimum,h.maximum
from sys.obj$ o,sys.hist_head$ h
where o.name='T1'
  and o.owner# = userenv('SCHEMAID')
  and o.obj#=h.obj# ;
--
set autotrace traceonly explain
select * from t1 where a > 1e30 ;
select * from t1 where a > 1e37 ;
select * from t1 where a < 1e30 ;
select * from t1 where a < 1e37 ;
select * from t1 where a >= 1e30 ;
select * from t1 where a >= 1e37 ;
select * from t1 where a <= 1e30 ;
select * from t1 where a <= 1e37 ;

This code sets the columns statistics to VDV and examines the consequences.
This is what we get:

LOWVAL                         HIVAL                             MINIMUM    MAXIMUM
------------------------------ ------------------------------ ---------- ----------
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF          0 1.3292E+36
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF          0 1.3292E+36
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF          0 1.3292E+36
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF          0 1.3292E+36
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF          0 1.3292E+36
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF          0 1.3292E+36
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF          0 1.3292E+36
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF          0 1.3292E+36
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF          0 1.3292E+36
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF          0 1.3292E+36
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF          0 1.3292E+36
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF          0 1.3292E+36
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF          0 1.3292E+36
13 rows selected.

SQL> --
SQL> set autotrace traceonly explain
SQL> select * from t1 where a > 1e30 ;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 20000 |  7812K|   223   (2)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 20000 |  7812K|   223   (2)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A">1e30)

SQL> select * from t1 where a > 1e37 ;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |     1 |   400 |   222   (1)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   |     1 |   400 |   222   (1)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A">1e37)

SQL> select * from t1 where a < 1e30 ;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 10000 |  3906K|   222   (1)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 10000 |  3906K|   222   (1)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A"<1e30) SQL> select * from t1 where a < 1e37 ;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 20000 |  7812K|   223   (2)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 20000 |  7812K|   223   (2)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A"<1e37) SQL> select * from t1 where a >= 1e30 ;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 20000 |  7812K|   223   (2)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 20000 |  7812K|   223   (2)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A">=1e30)

SQL> select * from t1 where a >= 1e37 ;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      |     1 |   400 |   222   (1)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   |     1 |   400 |   222   (1)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A">=1e37)

SQL> select * from t1 where a <= 1e30 ;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 10000 |  3906K|   222   (1)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 10000 |  3906K|   222   (1)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A"<=1e30) SQL> select * from t1 where a <= 1e37 ;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 20000 |  7812K|   223   (2)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 20000 |  7812K|   223   (2)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A"<=1e37)
 

You can see that cardinality estimates are badly affected when very large values are supplied in the predicates. We can get much more intelligible - and consistent – results by setting NOVALS ourselves. This is what happens when you uncomment the line labeled ‘Special Fix’ above.

LOWVAL                         HIVAL                             MINIMUM    MAXIMUM
------------------------------ ------------------------------ ---------- ----------
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF -9.90E+125 9.900E+125
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF -9.90E+125 9.900E+125
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF -9.90E+125 9.900E+125
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF -9.90E+125 9.900E+125
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF -9.90E+125 9.900E+125
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF -9.90E+125 9.900E+125
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF -9.90E+125 9.900E+125
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF -9.90E+125 9.900E+125
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF -9.90E+125 9.900E+125
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF -9.90E+125 9.900E+125
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF -9.90E+125 9.900E+125
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF -9.90E+125 9.900E+125
000000000000000000000000000000 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF -9.90E+125 9.900E+125
13 rows selected.

SQL> --
SQL> set autotrace traceonly explain
SQL> select * from t1 where a > 1e30 ;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 10000 |  3906K|   222   (1)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 10000 |  3906K|   222   (1)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A">1e30)

SQL> select * from t1 where a > 1e37 ;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 10000 |  3906K|   222   (1)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 10000 |  3906K|   222   (1)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A">1e37)

SQL> select * from t1 where a < 1e30 ;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 10000 |  3906K|   222   (1)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 10000 |  3906K|   222   (1)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A"<1e30) SQL> select * from t1 where a < 1e37 ;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 10000 |  3906K|   222   (1)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 10000 |  3906K|   222   (1)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A"<1e37) SQL> select * from t1 where a >= 1e30 ;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 20000 |  7812K|   223   (2)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 20000 |  7812K|   223   (2)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A">=1e30)

SQL> select * from t1 where a >= 1e37 ;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 20000 |  7812K|   223   (2)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 20000 |  7812K|   223   (2)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A">=1e37)

SQL> select * from t1 where a <= 1e30 ;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 20000 |  7812K|   223   (2)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 20000 |  7812K|   223   (2)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A"<=1e30) SQL> select * from t1 where a <= 1e37 ;

Execution Plan
----------------------------------------------------------
Plan hash value: 3617692013

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 20000 |  7812K|   223   (2)| 00:00:03 |
|*  1 |  TABLE ACCESS FULL| T1   | 20000 |  7812K|   223   (2)| 00:00:03 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("A"<=1e37)

This entry is a bit different in that, although the material is largely technical,it does indeed reflect some personal experiences.  Apologies to those of you who find the change of tone disappointing; it is not a sign of things to come.

The first, and most important, thing I have to discuss about this years conference concerns a question Wolfgang Breitling asked me during my “Stabilising Statistics” talk.  This presentation (material available for download here) focussed on improving execution plan stability without the use of stored outlines or SQL baselines.  One of the items I discussed was how execution plans changed as a result of bind variable values moving beyond the minimum and maximum values recorded in column statistics.  This question was of considerable interest to Wolfgang as he had himself presented a paper in 2009 on the same subject entitled “Seeding Statistics”.

Wolfgang’s original technique was to set the number of distinct values to 2 and the minimum and maximum values the same!  Of course this cannot happen in real life: if the minimum and maximum are the same then there is only one value.  Wolfgang reported that he got good, stable plans with this approach but my experiments suggested that setting the minimum value of the column to a very low value and the maximum to a very high value achieved better results.

Wolfgang’s question was this:  “What happens if you set the minimum and maximum values to NULL”?  Well, I knew that I had tried it and that the results were not satisfactory but I didn’t recall the specifics of my incomplete analysis.  I promised to blog an answer.  Well it turns out that the topic is quite complicated and Wolfgang and I have collaborated on some research over the holiday period and I will publish the results soon.

The second experience I wanted to share with you was an enlightening conversation I had with a few new friends in the pub.  The topic was Oracle security.  This important topic is something that I find rather dry and have not thought about as much as other aspects of Oracle database technology – such as performance.  I was talking to Martin Nash, Neil Johnson, Ben Thompson and others.  I suggested that if your data was sensitive in nature you had to buy Database Vault as DBAs would otherwise be able to see your data.  It was reminded that privileges such as ALTER DATABASE, ALTER USER, and the SELECT_CATALOG role did not allow a DBA to read any business data at all.  In fact, as long as you control the rare occasions when a database needs to be bounced (that does require SYSDBA) your DBAs should be able to do most if not all of their work without having access to any business data!  Limiting DBA privileges may not be as complete a solution to role segregation as Database Vault but when I thought about it I realised that it certainly goes further than I had previously realised.

Last but not least I want to mention the other event that I was involved in at the conference:  a debate with Jonathan Lewis entitled “Does Oracle Ignore Hints”?  One thing that I learned from Jonathan during the preparation for my debate was that the technical analysis in this entry was completely wrong.  I have posted a correction here.