Syed Jaffar

Subscribe to Syed Jaffar feed
Whatever topic has been discussed on this blog is my own finding and views, not necessary match with others. I strongly recommend you to do a test before you implement the piece of advice given at my blog.The Human Flyhttp://www.blogger.com/profile/03489518270084955004noreply@blogger.comBlogger298125
Updated: 3 days 23 hours ago

Monitoring & Troubleshooting Oracle Cloud at Customer

Fri, 2019-05-31 08:33
The prime advantage of cloud at customer is to deliver all cloud benefits at your data center. Oracle cloud at customer provides the same. When Oracle cloud at customer is chosen, it is Oracle who is responsible to install, configure and manage the software and hardware required to run Oracle cloud at customer. However, customers are responsible for monitoring and troubleshooting resources instantiate on Oracle cloud at customer.

Customers are required to understand the difference between system and user-space monitoring and the tools required. The Oracle cloud at customer subscription consists of the below components:


  • Hardware and Software
  • Control panel software
  • The Oracle Advanced Support Gateway (OASW)
  • The Oracle Cloud Service

System monitoring vs User Space Monitoring

Typically, Oracle cloud at customer is monitored at two level:
  1. System
  2. User space
Oracle monitors the system and the customer monitors the user space.



























The systems or machine resources, such as : hardware, control panel and cloud services on Oracle cloud at Customer are managed by Oracle remotely using Oracle Advanced Gateway. The OAGW is only used and accessible to Oracle authorized personnel.

The user space components consists the below:


  • Oracle Cloud accounts
  • VMs instances on IaaS or PaaS
  • DB that are provisioned within the PaaS subscription
  • Applications (Oracle or any third-party)
Oracle manages the following hardware and software components:
  • Ethernet switches
  • Power Supplies
  • Exadata Storage Servers
  • Hypervisor running on the physical servers
Customers can assign administrators to manage cloud accounts. Customers also are free to use any external monitoring agents to monitor user-space components.


Why Oracle Cloud @ Customer is a good option?

Thu, 2019-05-30 03:53
One of the major concerns moving over cloud is the security for most of the organizations. Though cloud concept is around for quite sometime, still, a majority of customers are concerned about putting their data over cloud. To gain the confidence and at the same to take full advantage of Cloud technologies, various Cloud vendors started offering cloud at customer solutions. In this blog spot, am going to discuss about Oracle cloud at customer solutions, its advantages , subscription model etc.

Oracle Cloud at Customer delivers full advantages of cloud technologies at your data center. You subscribe hardware and software together when you go for cloud at customer option. Though Oracle does the initial setup, configuration and day-to-day system management, you still have all the advantages of security, network of your data center.

Typically, the cloud at customer option consist of the following:


  • The hardware required to run Cloud at customer
  • Control panel software
  • The Oracle Advanced Support Gateway
  • Oracle Cloud services
 As a customer, your responsibility involves managing cloud account and subscribed services. At any time, you can check your account balance and your current Oracle Cloud at Customer service usage. It is also possible that you can view your usage by region, by service, or by a specific time period.
To check your account balance and usage, Oracle recommends that you sign in to your Oracle Cloud Account in an Oracle data region. From there, you can view your overall account usage and Universal Credits balance. 

In nutshell, cloud at customer brings the cloud solutions to your data center, where you can apply all the rules of your data centre while taking the full advantages of cloud solutions.

Network design for Oracle Cloud Infrastructure

Tue, 2019-05-28 15:48
Assuming, you are planning to migrate your resources from Oracle Cloud Infrastructure Compute classic environment to Oracle Cloud Infrastructure, this blog post explains the details of network design for Cloud Infrastructure environment. It's important to understand and map the network design and details from both environments.

Cloud Inrastructure Compute Classic network has IP Networks and Shared Network model. On other hand, Cloud Infrastructure has Virtual Cloud Networks (VCNs) , Subnets, Availability Domain network model.

Before migration, you must map the network resources between the environments. Source -> Target:
Shared network -> VCN, IP Network -> IP Network, VPN -> IPSec VPN and Fast Connect classic -> FastConnect.

Consider creating below listed network elements in Oracle Cloud Infrastructure:

  • VCN and Subnet CIDR Prefixes
  • DNS Names 
Use the below procedure to configure cloud network for Cloud Infrastructure environment:


  1. Create one or more VCNs.
  2. Create an Internet gateway and/or NAT gateway. An Internet gateway is a virtual router that allows resources in a public subnet direct access the public Internet. A NAT gateway allows resources that don't have a public IP addresses to access the Internet, without exposing those resources to incoming traffic from the Internet.
  3. Configure a service gateway, if required. A service gateway provides a path for private network traffic between your VCN and a public Oracle service such as Oracle Cloud Infrastructure Object Storage.
  4. Create one or more subnets in each VCN.
  5. Configure local peering gateways between VCNs, if required.
  6. Configure security lists, security rules, and route tables for each subnet.

Migrating Oracle Cloud Infrastructure Classic Workloads to Oracle Cloud Infrastructure - Migration Tools

Tue, 2019-05-28 03:22
If you are planning to migrate your resources from Oracle Cloud Infrastructure Classic to Oracle Cloud Infrastructure, Oracle provides variety of tools to achieve this. In this blog post will discuss some of thetools which can be used to migrate Oracle Cloud Infrastructure Classic Workload resources to Oracle Cloud Infrastructure. 

The tools below can be used to identify resources from Oracle cloud infrastructure Classic environments and to migrate to Oracle Cloud Infrastructure tenancy. Using these tools, one can setup required network, and migrate VMs and block storage volumes over the targeted systems.

Tools for migrating infrastructure resources : Compute, VMs and Block Storage

  • Oracle Cloud Infrastructure Classic Discovery and Translation Tool: as the name explains, it is a discovery tool, which assist discovering the resources of different resources in your Cloud Infrastructure Classic environments, such as : compute Classic, Object Storage classic, load balancing classic account. Additionally, it is capable of reporting the items in the specified environment, list of VMs in the source environment and also networking information of the source system.
  • Oracle Cloud Infrastructure Classic VM and Block Storage Tool: This tool automates the process of migrating VMs and Block Storage over to the target environment. 
  • Oracle Cloud Infrastructure Classic Block Volume Backup and Restore Tool: This tool used to migrate your remote snapshot of storage volumes as well as scheduled backups. 

Tools for migrating databases

To migrate databases to Oracle cloud infrastructure, you can use Oracle Cloud Infrastructure Class Database Migration tool. This tool uses Oracle RMAN to create backup and restore the database over the targeted system.

Alternatively, Oracle Data Guard solution also can be used to migrate single or RAC databases to Oracle Cloud Infrastructure.

Tools for migrating Object Storage

  • rclone command is used to migrate your object storage data f you don't use the Oracle Cloud Infrastructure Storage Software Appliance.
  • If Oracle Cloud Infrastructure Storage Software Appliance is used to store your object data, then you can migrate your data to your Oracle Cloud Infrastructure Object Storage account by using the Oracle Cloud Infrastructure Storage Gateway.





How to change Remote Trail Location at Source– Golden Gate 12c

Sun, 2019-05-26 02:29

There is a requirement to modify the remote trail location because of typo in the mount point. To change the remote trail location of the Golden Gate, first verify the extract trail information using the command below. 

GGSCI (hie-p-ggate) 48> INFO EXTTRAIL

       Extract Trail: /golgengate/ggs_home/dirdat/test/ru
        Seqno Length: 6
   Flip Seqno Length: yes
             Extract: DPMP1
               Seqno: 0
                 RBA: 0
           File Size: 500M

       Extract Trail: D:\app\oracle\product\ogg_1\lt
        Seqno Length: 6
   Flip Seqno Length: yes
             Extract: EXT1
               Seqno: 0
                 RBA: 0
           File Size: 100M
GGSCI (hie-p-ggate) 49>

First delete the current configuration of remote trail. 

GGSCI (hie-p-ggate) 50> delete rmttrail /golgengate/ggs_home/dirdat/test/ru, extract dpmp1
Deleting extract trail /golgengate/ggs_home/dirdat/test/ru for extract DPMP1

After removing, then add the remote trail again with the new location. 


GGSCI (hie-p-ggate) 51> add rmttrail /goldengate/ggs_home/dirdat/test/ru, extract dpmp1
RMTTRAIL added.


After deleting and adding now we can see the extract trail for the target with new location. 

GGSCI (hie-p-ggate) 52> INFO EXTTRAIL

       Extract Trail: /goldengate/ggs_home/dirdat/test/ru
        Seqno Length: 6
   Flip Seqno Length: yes
             Extract: DPMP1
               Seqno: 0
                 RBA: 0
           File Size: 500M

       Extract Trail: D:\app\oracle\product\ogg_1\lt
        Seqno Length: 6
   Flip Seqno Length: yes
             Extract: EXT1
               Seqno: 0
                 RBA: 0
           File Size: 100M



GGSCI (hie-p-ggate) 53>


Oracle Database Upgrade made easy with AutoUpgrade utility

Tue, 2019-05-21 03:59
Upgrading Oracle database is undoubtedly a daunting task which requires a careful study, planning and execution to prevent any potential post upgrade shortcomings. Since Oracle determined to release a new Oracle version every year, at some-point-of-time, we all should focus on upgrading databases quite often than we used to perform earlier.

Thanks to AutoUpgrade tool(utility), available with MOS Doc ID : 2485457.1, automates all upgrade process procedure without much human intervention or inputs. For latest AutoUpgrade utility version, always refer the MOD note and download. Though, with 12.2(DBJAN2019RU), 18.5 and 19.3, the AutoUpgrade utility available by default under the oracle home.

AutoUpgrade is a command-line tool which can be used to upgrade one or many oracle databases with one command and a single configuration file. This utility automates upgrade process, such as, pre-upgrade tasks, performs automated fix-ups, perform database upgrade and run through post upgrade tasks as well. This saves a huge time and money when upgrading hundreds of databases in any environment.



I am pretty sure AutoUpgrade make DBA's like easier when it comes to Oracle database upgrade. Have fun and test the capabilities of the tool.



References:
MOS Doc ID : AutoUpgrade Tool (Doc ID 2485457.1)
https://mikedietrichde.com/2019/04/29/the-new-autoupgrade-utility-in-oracle-19c/
https://docs.oracle.com/en/database/oracle/oracle-database/19/upgrd/about-oracle-database-autoupgrade.html#GUID-3FCFB2A6-4617-4783-828A-41BD635FC88C

Whats in Exadata X8 Server 19.2 Exadata Database Machine?

Sun, 2019-04-14 05:01
This blog post quickly scan through the new features of Exadata Database Machine in 19.2 and also hardware capacity and change in Exadata X8 server.


  • Exadata Database Machine Software 19.2.0, supports Exadata X8-2 and X8-8 hardware
  • Changes in IORM's flashcachesize and Disk I/O limits attributes
  • To control the cost of Exadata storage, X8 introduced a new configuration, Exadata Storage Extended (XT)
  • The XT model comes with 14TB hard drives with HCC compression capacity
  • The XT model doesn't have flash drive
  • The lower cost storage option comes with one CPU, less memory and without the core feature of SQL offloading
  • Exadata X8 server has the below hardware capacity per rack:
    • Limit of 912 CPU core and 28 TB memory
    • 2-19 database servers
    • 3-18 cell storage
    • 920 TB of RAW flash capacity
    • 3 PB of RAW disk capacity

References:

https://www.oracle.com/a/ocom/docs/engineered-systems/exadata/exadata-x8-2-ds.pdf
https://docs.oracle.com/en/engineered-systems/exadata-database-machine/dbmso/whats-new-oracle-exadata-database-machine-19.2.html#GUID-B2A6BEAD-873A-4524-82EF-CDF53098820B

Oracle 19c Automatic Indexing - How well it's understood?

Sun, 2019-03-10 13:05
I wrote an article about Oracle 19c Automatic Indexing. Go to the below link for more details:

https://medium.com/@sjaffarhussain/oracle-19c-automatic-indexing-how-well-its-understood-1798d1b3d097

Oracle 19c ASMCA interface

Mon, 2019-02-25 02:08
This blog post will walk through some of very cool screen shots of 19c ASMCA interface. I have no exposure with 18c ASMCA, but, the landing page of 19c ASMCA is really cool. Here are the screenshot for you:

./asmca



ASM Instance Management


Disk group Management
















DG attributes



 root setup

What's new in 19c - Part III (Data Guard)

Sun, 2019-02-24 06:36
Business continuity (Disaster Recovery) has become a key aspect of every business for a long time now. Oracle data guard is one of the best solutions for business critical applications running on  Oracle databases. From its inception, a lot has been enhanced in standby database functionality to meet the market demand.

This blog post is dedicated and focused on some key enhancements introduced in 19c Data Guard. Below are my hand-picked list of new features, which really got my attention:

Fast-Start-Failover (FSFO) in Observer-only Mode

Configuring FSFO was really a big debate for quite sometime in Oracle community. Some may recommend and some are not in favor of enabling FSFO. Personally, I was not in favor of this feature. Though the decision is lot depends on various factors.

FSFO can be configured in observe-only mode wit 19c (validate without real action), which allow DBAs to test an automatic failover configuration without actually causing any damage to the databases in production environment. When FSFO is configured in observer-only mode, no actual changes are made to the DG Broker settings, also doesn't require any application changes. And, when the conditions for FSFO are met, the DG Broker adds the messages to the observer log indicating that FSFO would have been initiated. This makes it easer to justify using FSFO to reduce the recovery time for real failover.

To enable FSFO in observer mode, use the below syntax:

DGMGRL> ENABLE FAST_START FAILOVER OBSERVER ONLY;

Automatic Flashback Standby

Prior to Oracle 19c, when flashback database or point-in-time operations are performed on a primary database, the underlying standby database needs to be put into same point-in-time as its primary database with a manual procedure (for example, FLASHBACK STANDBY DATABASE TO SCN resetlogs_change# - 2;). This functionality is automated in 19c. This new feature enables the standby database to be flashed back automatically whenever flashback database operation is triggered on the primary database. By automating this process, it drastically reduces the time & efforts and improves RTO.

So, when a primary database has any flashback database or point-in-time recovery operations, the standby automatically follow the primary database, and the MRP on standby database perform the following actions:

  • detect the new incarnation
  • flashback the standby to the exam time as its primary
  • restart the standby recovery and move the standby to the new branch of redo
** Note : Flashback operation success is subject to flashback data availability


Automatic flashback standby operation takes place when the database is opened in MOUNT state. If the standby database is open in READ ONLY mode, the error messages are recorded in the alert log and whenever standby database is restarted, the recovery process (MRP) automatically executes the flashback operation.

DML Operations on Active Data Guard

Performing DML operations on Active Data Guard was something long awaited. I remember, there are some application that needs to long an entry into the database whenever they connected to the database. This was causing many applications no to use with Data Guard, specially for testing.

So, it's here finally with Oracle 19c. Though Oracle doesn't recommend heavy DML operations on active standby database pridicting the performance impact on the primary database. But, it's good for applications that mostly read-applications with occasional DML executions.

To configure DML operations, set ADG_REDIRECT_DML init parameter to TRUE or execute the following SQL statement:

SQL> ALTER SESSION ENABLE ADG_REDIRECT_DML;

Subsequently, perform the DML operations:

SQL> INSERT INTO table VALUES (.......);

** The settings can be database or session level.

DML operations on active standby database are transparently redirected to the primary database upon setting the above configuration, including DML operations that are part of PL/SQL blocks. However, the active data guard session waits until the corresponding changes (DML) are shipped to and applied to the ADG standby database, while maintaining the read-consistency.

To redirect PL/SQL operation from active standby data guard database to primary database, configure the following:

SQL> ALTER SESSION ENABLE ADG_REDIRECT_PLSQL;

Subsequently, perform the PL/SQL operations :

SQL> CREATE OR REPLACE PROCEDURE ....;

Automatic outage resolution with Data Guard

One of the common scenarios of delayed redo transport and gap resolution on data guard is due to network hangs, disconnects, and disk I/O issue. With 19c, new DATA_GUARD_MAX_IO_TIME and DATA_GUARD_MAX_LONGIO_TIME parameters, DBA can tune the amount of wait time for those detection based on the user network and Disk I/O behavior. Data Guard has an internal mechanism to detect these hung processes and terminate them allowing the normal outage resolution to occur.

Stay tuned for more 19c new features.


What's new in 19c - Part II (Automatic Storage Management - ASM)

Wed, 2019-02-20 07:32
Not too many features to talk on 19c ASM. Below is my hand-pick features of 19c ASM for this blog post.


Automatic block corruption recovery 

With Oracle 19c, the CONTENT.CHECK disk group attribute on Exadata and cloud environment is set to true by default. During data copy operation, if Oracle ASM relocation process detects block corruption, it perform automatic block corruption recovery by replacing the corrupted blocks with an uncorrupted mirror copy, if one is avialble.


Parity Protected Files Support

The level of data mirroring is controlled through ASM disk group REDUNDANCY attribute. When a two or three way of ASM mirroring is configured to a disk group to store write-once files, like archived logs and backup sets, a great way of space is wasted. To reduce the storage overahead to such file types, ASM now introduced PARITY value with REDUNDANCY file type property. The PARITY value specifies single parity for redundancy. Set the REDUNDANCY settings to PARITY to enable this feature.

The redundancy of a file  can be modified after its creation. When the property is changed from HIGH, NORMAL or UNPROTECTED to PARITY, only the files created after this change will have impact, while the existing files doesn't have any impact.

A few enhancements are done in Oracle ACFS, Oracle ADVM and ACFS replication. Refer 19c ASM new features for more details.


** Leaf nodes are de-supported as part of Oracle Flex Cluster architecture from 19c.


Whats new in 19c - Part I (Grid Infrastructure)

Tue, 2019-02-19 03:41
Every new Oracle release comes with bundle of new features and enhancements. Though not every new feature is really needed to everyone, there are few new features that worth considering. As part of 19c new features article series, this post is about the new features introduced in Grid Infrastructure. This blog post focuses on some real useful GI features with deprecated and de-supported features in 19.2.

Dry-run to validate Cluster upgrade readiness

Whether it's a new installation or upgrade from previous version to latest version, system readiness is the key factor for success. With 19c, Cluster upgrade can have a dry-run to ensure the system readiness without actually performing the upgrade of the cluster. To determine if the system is ready for the upgrade, run the upgrade in dry-run mode. During the dry-run upgrade, you can click the Help button on the installer page to understand what is being done or asked.

Use the command below from the 19c binaries home to run the cluster upgrade in Dry-run mode:

$ gridSetup.sh -dryRunForUpgrade

Once you run through with all the interactive screens for dry-run, check the gridSetupActions<timestamp>.log file for errors and fix them for real upgrade run.

Multiple ASMBn

It is a common practice to have multiple disk groups in a RAC environment. It is also possible to have some disk groups in MOUNT state and some disk groups in DISMOUNT state on a DB node. However, when a db instance on a node try to communicate (startup) with the DISMOUNT disk group will throw errors.

Multiple ASMB project allows for the database to use DISK GROUPS across multiple ASM instances simultaneously.  This enhancement provides the HA to RAC stack by allowing DB to use multiple disk groups even if a given ASM instance happens to have some DISMOUNT disk groups.

AddNode and Cloning with Installer Wizard

Adding a new node and the functionality of installing a gold image (cloning) is simplified and made easy in 19c. Adding new node and Cloning homes now directly available with Installer Wizard, you no longer need to use add node.sh and clone.pl scripts. These commands will be depreciated in the upcoming releases.

In the upcoming blog, I will discuss about ASM 19c features.




Oracle 19c and my favorite list

Thu, 2019-02-14 05:37
Today (14-Feb-2019) Oracle officially released the 19c docs and Oracle Database 19c for Exadata through edelivery channel. Since the news is out, Oracle community is busy talking about 19c availability and sharing articles about 19c etc.

I spent a little amount of time to scan through some of really useful features of 19c for DBAs, and here is my list:
  • Availability
    • Simplified DB parameter management in Broker Configuration
    • Flashback Standby DB when Primary DB is flashed back
    • Active Data Guard DML Redirection
    • New parameter for tuning automatic outage resolution with DG
  • Data Warehousing
    • SQL Diagnostic and Repair Enhancements
    • Automatic Indexing 
    • Performance Enhancement for in-memory external tables
    • Real-Time Statistics
    • High Frequency Automatic Optimizer Statistics Collection
  • Automated install, config and patch
  • Automated upgrade, migration and utilities
  • Performance
    • SQL Quarantine 
    • Real-time monitoring for Developers
    • Workload capture and Replay in a PDB
  • RAC and Grid
    • Automated Transaction Draining for Oracle Grid Infrastructure Upgrades
    • Zero-downtime Oracle Grid Infrastructure Patching

I will start writing series of articles about my favorite Oracle 19c features. Stay tune.



References:
https://www.oracle.com/a/tech/docs/database19c-wp.pdf
https://docs.oracle.com/en/database/oracle/oracle-database/19/newft/new-features.html#GUID-06A15128-1172-48E5-8493-CD670B9E57DC
https://medium.com/oracledevs/oracle-database-19c-now-available-on-oracle-exadata-9b57963e2c89

ORA-600 [ossnet_assign_msgid_1] on Exadata

Mon, 2019-02-04 06:10
On a Exadata system with Oracle v12.1, a MERGE statement with parallelism was frequently failing with below ORA errors:

                   ORA-12805: parallel query server died unexpectedly
        ORA-06512

A quick look in the alert.log, an ORA-600 is noticed.

ORA-00600: internal error code, arguments: [ossnet_assign_msgid_1], [],[ ] 

The best and easy way to diagnose any ORA-600 errors is to utilize the ORA-600 tool available on MOS.

In our case, with large hash join, the following MOS note helped to fix the issue:

On Exadata Systems large hash joins can fail with ORA-600 [OSSNET_ASSIGN_MSGID_1] (Doc ID 2254344.1)

Cause:
On Exadata Systems large hash joins can fail with ORA-600 [OSSNET_ASSIGN_MSGID_1] and the root cause if often a too small default value  for _smm_auto_min_io_size  and  _smm_auto_max_io_size'

and the workaround to fix the issue is to set the following underscore (_) parameters:

_smm_auto_max_io_size = 2048
_smm_auto_min_io_size = 256

In some cases, the below MOS notes helps to fix ORA-600 [ossnet_assign_msgid_1] error.

ORA-600 [ossnet_assign_msgid_1] (Doc ID 1522389.1)
Bug 14512766 : ORA-600 [OSSNET_ASSIGN_MSGID_1] DURING RMAN CONVERSION

Automated Cell Maintenance

Wed, 2019-01-23 05:20
One of the key actives for a DBA is to well maintain the database servers and Oracle environments. In a complex Oracle environment, managing and maintaining file system space plays a very crucial role. When a FS, where Oracle binaries are stored,  runs out of space, it could lead to some sort of consequences and some situations it can cause service interruption.

One of the routine actives for a DBA in a very busy system is to maintain the FS space by regularly purging or cleaning the old log and trace files. Some DBAs perform these activities through a schedule job. However, Oracle does introduced an auto maintain jobs. For example, in a cluster environment, the logs are maintained in terms of size as well as retention of the historical copies. On Exadata too Oracle has automated the Cell maintenance in place.

In this blog post, we will run through some of useful information about automated cell maintenance activities.

The Management Server (MS) component carries the responsibility of auto space management. For example, when there is a shortage of space in ADR, the MS deletes the files as per below default policy:


  • Files which are older than 7 days in ADR and LOG_HOME directories
  • alert.log will be renamed once it reaches to 10MB, and the historical files are kept for 7 days
  • Upon 80% of FS utilization, the MS triggers the deletion policy for / (root) and /var/log/oracle directories
  • Similar, the deletion policy will be activated when the /opt/filesystem 90% utilized
  • Alerts are cleared based on the criteria and policies

The default retention policy is set to 7 days. If you want to modify the default behavior, you will have to change the metricHistoryDays and dragHistoryDays attributes with ALTER CELL command.

Read the below Oracle document for more insights about auto cell maintenance tasks.

https://docs.oracle.com/en/engineered-systems/exadata-database-machine/sagug/exadata-storage-server-configuring.html#GUID-EACAE5AF-A89D-4A3D-9CC1-A99D6E6FE46E

Automated Cloud Scale Performance Monitoring capabilities with Exadata Software version 19.1

Tue, 2019-01-22 05:11
Starting with v12.2, Oracle Autonomous Health Framework (AHF) multiple components work together 24x7 autonomously to keep the database system healthy and reduces human intervention & reaction time utilizing machine learning technologies .

There is no doubt that Exadata Database Machine delivers extreme performance for all sorts of workload. However, diagnosing critical performance issues still needs some manual work and human intervention to identify root causes. This blog post highlights a new autonomous performance enhancement introduced with Exadata system software v 19.1.

Exadata software Release 19.1 comes with an automated, cloud-scale performance monitoring for a wide-range of sub-systems, such as: CPU, Memory, File System, I/O and network. This feature built with the combination of years of real-world performance triaging experience by Oracle Support, industry best practices and Artificial intelligence (AI) technologies. This feature simplifies root cause analysis without much human intervention. It can automatically detect runtime critical performance issues and also figure out the root cause without human intervention.

Taking a few real-world scenarios, as a DBA, we all have come across on many occasions where a spinning process on a database server eating up all system resources causing complete database death (poor performance). With this enhancement, Exadata System Software automatically identifies the exact process that is causing spinning and generates an alert with root cause analysis. Another typical example will be automatically detecting the misconfiguration of huge pages settings on the server and sending alerts. When how a server and perform badly if the huge pages setting is right on the system.

No additional configuration and special skill set is required for this. Management Server (MS) is responsible to perform these activities. All you need is have Exadata software version 19.1 or higher, and configure your alerts on the servers.

For more details, read the oracle documentation.

https://docs.oracle.com/en/engineered-systems/exadata-database-machine/dbmso/whats-new-oracle-exadata-database-machine-19.1.0.html#GUID-B3DE3C62-278A-48E3-889A-22B9C84B1413

Stay tuned and hunger for more Exadata software 19.1 new features.

How to manage DB and Cell servers remotely on Exadata with ExaCLI utility - (Part 1)

Sun, 2018-12-30 05:26
Whoever is working with Exadata or knew how to administrate Exadata major components like DB and Cell nodes, will certainly knows the use of CellCLI, dcli and DBMCLI utilities. This blog post is focused about managing database and cell servers remotely using the ExCLI and ExadCLI utilities.

Simply put, ExaCLI is a command line tool which comes by default on database & cell nodes that provides the capabilities of remote management for database and cell servers on the Exadata. Unlike CellCLI only runs on cell servers and DBMCLI runs on only DB nodes, the ExaCLI can manage database or cell servers remotely.

There are two key advantages of using the ExaCLI: 1) Its useful when you can't get SSH connectivity and root user credentials to connect DB or Cell nodes. 2) With Exadata at customer or cloud, customers won't get SSH and root user access for CellCLI and DBMCLI. So, the ExaCLI could be handy to access the servers.

ExaCLI works with the non-system (default) users on the DB or cell serers. Therefore, you must create a role based user in order use the utility to connect to a DB or cell server. The utility is found under /user/local/sbin directory.



Creating users for ExaCLI




Note : not all CellCLI commands are compatible on ExaCLI. Below sections givens an overview about the limitations:


Below are some of the examples demonstrate how to establish remote connectivity with DB or cell servers using ExaCLI:




In the next blog post, you will learn how to use the Exadcli utility execute the commands on a set of DB or cell nodes at once.


References:

https://docs.oracle.com/en/engineered-systems/exadata-database-machine/dbmmn/exadcli.html#GUID-1C738F05-2A69-4B75-BB1E-B578C9081487

https://docs.oracle.com/en/engineered-systems/exadata-database-machine/dbmmn/exacli.html#GUID-3D57CDC3-561C-48CE-AB1E-0279E6D24DAE

Exadata Cloud Machine - Hardware capacity

Sun, 2018-11-18 04:13
Everyone who is aware and utilizes Exadata Database Machine is certainly knew the performance it can deliver. I have involved in many Exadata migration projects, and witnessed how customers gained the database performance and satisfied post migration. I am not talking about the cost, the need etc., as a technical guy, I knew the capabilities of the box and how it can benefit customers to fulfill their need and future demand.

We all knew about Cloud technologies, how every software company and organization trying to race with the trend and need of cloud technologies. In some countries, the cloud adoption is bit slower compare to the other part of the world. But, gradually majority of the companies would be adopting cloud technologies, this is for sure. Certainly, cloud has its own share of advantages and disadvantages. Whoever utilizes it smartly, can gain much flexibility and benefits.

To ensure and meet customers demand to have Exadata availability on cloud, Oracle started Exadata Cloud services offering to facilitate Exadata machine on cloud. Still, some organization couldn't adopt cloud due to industry regulations, corporate policies, security compliance etc. Therefore, Oracle announced Exadata Cloud Machine availability. With this model, customers who want to have cloud on-premises with Exadata hardware, can go for this model.

I would like to highlight the hardware capabilities that Exadata Could Machine (ExaCM) offers.


  • 40Gbps InfiniBand Networking
  • Ultra-fast NVMe Flash storage
  • Up to 257GB/sec Throughput
  • Up to 3.6 Million 8k I/Os per sec
  • 1/4 millisecond response time
  • Fastest Compute 
  • Fastest x86 processors
  • Large Memory Capacity - 720GB per compute node
  • Complete Redundancy
Soon will talk more about Exadata Cloud Machine migrations. Stayed tuned and hunger for knowledge.

Few Exadata MOS Docs to review

Wed, 2018-10-10 04:20
If you have MOS login credentials and managing Exadata database machines, below is the list of few MOS Doc which is worth reading:

  • 888828.1, "Exadata Database Machine and Exadata Storage Server Supported Versions"
  • 1070954.1, "Oracle Exadata Database Machine Exachk or HealthCheck"
  • 1353073.2, "Exadata Diagnostic Collection Guide"
  • 1500257.1, " Exadata Write-Back Flash Cache - FAQ"
  • 1553103.1, "dbnodeupdate.sh and dbserver.patch.zip: Updating Exadata Database Server Software using the DBNodeUpdate Utility and patchmgr"
  • 1589868.1, "Procedure to check for corrupted root file system on Exadata Storage Servers and Linux database servers"

(EX42) Flash disk failure may lead to ASM metadata corruption when using write-back flash cache

Mon, 2018-10-08 07:19
While reviewing the latest Exachk report on X5-2 machine, the following critical alrams were observed:



And details shows below description:


And the MOS Note : 1270094.1 explains the following:


According to MOS Doc: 2356460.1, the said behavior is due to a bug (27372426) which applies on Exa version 12.2.1.1.0 to 12.2.1.1.5 or 18.1.0.0.0 to 18.1.3.0.0.

Impact:

If you are running GI 11.2.0.4 or 12.1 with the above said Exa version, and  with FlashCache configured as Writeback mode, the following ORA error may encounter, during: ASM rebalancing operation, disk group mount, & disk group consistency checks, ASM review asm alert.log:

ORA-00600: internal error code, arguments: [kfdAuDealloc2]

WARNING: cache read a corrupt block: group=1(DATA) fn=381 indblk=27 disk=110 (DATA_CD_04_DM01CEL01)
ORA-15196: invalid ASM block header [kfc.c:26411] [endian_kfbh]

ORA-00600: internal error code, arguments: [kfrValAcd30]

ORA-00600: internal error code, arguments: [kfdAuPivotVec2], [kfCheckDG]

ERROR: file +DATADG1.3341.962251267: F3341 PX38530 => D55 A765853 => F1677
PX1647463: fnum mismatch
ERROR: file +DATADG1.3341.962251267: F3341 PX38531 => D15 A205431 => F3341
PX56068: xnum mismatch



Workaround:
To fix the bug, Following action plan needs to be applied:

1) Update the storage server to >=12.2.1.1.6 or >=18.1.4.0.0
2) Apply patch 27510959 and scan ASM metadata


Note :

The issues doesn't impact on GI 12.2 or whenever you have higher version of Exa software mentioned in this bug;
The bug also doesn't affect if the FlashCache mode is WriteThrough;

References:

Exadata Critical Issues (Doc ID 1270094.1)


(EX42) Flash disk failure may lead to ASM metadata corruption when using write-back flash cache (Doc ID 2356460.1)

Pages