Tuesday, August 14, 2012

Importing Ubuntu (non-Redhat-based) OSes into cobbler

I normally test my database stuff in CentOS which is why I automated the installation using cobbler. Importing CentOS into cobbler is no big deal. Neither is importing debian-based distributions like Ubuntu - if you know the one additional parameter that is needed.

In my first try, I didn't:
[root@infra01 ~]# cobbler import --name=Ubuntu12.04 --path=/mnt
task started: 2012-08-12_130430_import
task started (id=Media import, time=Sun Aug 12 13:04:30 2012)
Found a debian/ubuntu compatible signature: pool
adding distros
avoiding symlink loop
avoiding symlink loop
avoiding symlink loop
No distros imported, bailing out
!!! TASK FAILED !!!
It works without problems when adding the "breed" parameter:
cobbler import --name=Ubuntu12.04 --path=/mnt --breed=debian
Also see the wiki page on that topic.

Friday, August 10, 2012

Zombie Blackouts From The Past

Recently, I've been doing a cluster-wide maintenance and put all nodes in Full Blackout for that time (Enterprise Manager 11g).
When the blackout was stopped, for some reason older blackouts from as far back as 13 months ago started to activate themselves again on the agent side.
In the console, the affected targets just showed "Status Pending" - but it could be verified that the Blackouts were the root cause via issuing on the target host:

emctl status blackout

Though I haven't found the source of the problem yet, I used a one-liner to stop all those that I want to share for anyone having a similar problem. Doing them one by one just gets tiresome at a certain amount. And it's not the first time I encountered those.

emctl status blackout | grep Blackoutname | awk '{sub(/Blackoutname = /,"",$0); print "emctl stop blackout \""$0"\""}' | bash

Also might come in handy if you'd just like to get rid of several blackouts on a host at once.

Will update once/if I find the root cause and a solution.

Thursday, July 12, 2012

Enterprise Manager SQL Script Job errors

I use the Enterprise Manager Job System very often, such a convenient way of getting data from different database sources (even more useful via emcli into a common output location).

Some errors that come up doing that frequently include:

SQL*Plus: Release 11.2.0.3.0 Production on Thu Jul 12 18:57:32 2012

Copyright (c) 1982, 2011, Oracle. All rights reserved.

SQL> SQL> SQL> SQL> ERROR:
ORA-01033: ORACLE initialization or shutdown in progress
Process ID: 0
Session ID: 0 Serial number: 0


SP2-0584: EXIT variable "SQLCODE" was non-numeric
Usage: { EXIT | QUIT } [ SUCCESS | FAILURE | WARNING | n |
| : ] [ COMMIT | ROLLBACK ]

Error from SQLPLUS: 1
This is a fairly easy one - trying to query a Data Guard standby database which is of course not open as user who is not SYSDBA.
Just don't execute the job on standby database targets or - if the task you want to be done can be done in mount state normally - execute it as SYSDBA user.

Next one is a little more tricky:

SQL*Plus: Release 11.2.0.3.0 Production on Thu Jul 12 18:57:32 2012

Copyright (c) 1982, 2011, Oracle. All rights reserved.

SQL> SQL> SQL> SQL> ERROR:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0


SP2-0584: EXIT variable "SQLCODE" was non-numeric
Usage: { EXIT | QUIT } [ SUCCESS | FAILURE | WARNING | n |
| : ] [ COMMIT | ROLLBACK ]

Error from SQLPLUS: 1

You'll get this one when your database target is registered to the wrong agent (e.g. in a failover environment). The job system does not connect to the database directly by any kind of sqlnet usage - it logs on to the server and executes the local sqlplus binary. That's also why you have to specify host credentials for an SQL Script job, in case you ever wondered.

emcli relocate (also described here) to the correct agent solves that behavior.

Tuesday, June 12, 2012

Installing a VirtualBox Extension Pack

A lot of hits from search engines to this blog have keywords about installing a VirtualBox Extension Pack, seems people have problems doing that.

I showed how to do it in my very first post about installing VirtualBox on Fedora, all via command line.

That is still a valid approach as you can see here (this time on Windows 7):

C:\Users\robert>"C:\Program Files\Oracle\VirtualBox\VBoxManage" extpack install C:\Users\robert\Downloads\Oracle_VM_VirtualBox_Extension_Pack-4.1.16-78094.vbox-extpack
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Successfully installed "Oracle VM VirtualBox Extension Pack".

Be sure to enclose the path to the VBoxManage executable and the extension pack file in quotes in case there are any spaces in the path.

Where to get the vbox-extpack file in the first place? From the Download page of virtualbox.org.

Installing VirtualBox or an Extension Pack for it however is nothing one typically does every day that would need to be scripted. Here's how to do it via the GUI:

  1. Click File - Preferences.
  2. Go to Extensions and click the icon on the right for adding packages.
  3. Navigate to the path where the Extension Pack was downloaded and open it.
  4. Confirm that you want to install the Pack.
  5. Carefully read the License (*cough*...) and agree... You'll at least have to scroll to the bottom for the button to be clickable and risk selling your soul.
  6. Get confirmation for the successful operation.

Monday, June 11, 2012

Performance Tab, Memory Advisor slow or hanging in Enterprise Manager

When you're responsible for the Oracle Enterprise Manager, you'll likely hear "it's slow!" a lot. And most of the time, users mean the Performance Tab or anything related to that.

Had that today again when someone tried to access the Memory Advisor of a database but clicking the link just resulted in an eternity of page loading and finally an error message as well as a warning alert for the "/secFarm_GCDomain/GCDomain/EMGC_OMS1" target along the lines
"Errors in directory: /u00/app/oracle/product/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/adr/diag/ofm/GCDomain/EMGC_OMS1/incident/incdir_n (incident=n):~null"
- generic as can be, also the info inside the incident directory.

Checking the database host, I noticed a process (remote connecting to the database in question) taking lots of CPU. The respective DB session that process represented was executing for minutes already:

select component, final_size, end_time
from v$sga_resize_ops
where initial_size <> final_size order by end_time;

The Enterprise Manager queries that when building the page for the memory advisor.

Problem here was that the fixed object statistics and the dictionary statistics were not collected after a migration to 11g so all the internal views - where EM gets its live information from for those functionalities, it's not all stored in the repository - performed badly.

Everything was just fine after making up for that:

exec dbms_stats.gather_fixed_objects_stats();
exec dbms_stats.gather_dictionary_stats();
Lessons learned:
  1. "Enterprise Manager is slow" does not always mean "Enterprise Manager is slow"
  2. Have your dictionary and fixed objects statisics up to date
  3. Failing to do so can generate some serious load on your system - just by trying to use EM

Monday, May 21, 2012

Here's an odd one: Do not comment the line before the last at object creation... when using Quest Toad. Or Oracle SQL Developer. Possibly others.

Why not? Datapump (more specific: impdp) might create problematic sql scripts as a result.

Validation:

Create the user:

CREATE USER testuser
IDENTIFIED BY testuser
DEFAULT TABLESPACE USERS
TEMPORARY TABLESPACE TEMP
PROFILE DEFAULT
ACCOUNT UNLOCK;
GRANT CONNECT TO testuser;
ALTER USER testuser DEFAULT ROLE ALL;
GRANT CREATE VIEW TO testuser;
ALTER USER testuser QUOTA UNLIMITED ON USERS;

Create 2 views, the first one having a comment at the end before the line with the semicolon:

create or replace force view testuser.test (v_col) as
select object_name from user_objects
where object_type='VIEW' -- added 20120521 by Robert Hanuschke
;

create or replace force view testuser.test2 (v_col) as
select object_name from user_objects
where object_type='VIEW'
;

Export the user:

expdp system schemas=testuser directory=DATA_PUMP_DIR dumpfile=testuser.dmp

Create the sqlfile using impdp:

impdp system schemas=testuser dumpfile=testuser.dmp directory=DATA_PUMP_DIR sqlfile=test.sql

Have a look at the View creation part of the SQL script being generated in the DATA_PUMP_DIR:

-- new object type path: SCHEMA_EXPORT/VIEW/VIEW
CREATE FORCE VIEW "TESTUSER"."TEST" ("V_COL") AS
select object_name from user_objects
where object_type='VIEW' -- added 20120521 by Robert Hanuschke;
CREATE FORCE VIEW "TESTUSER"."TEST2" ("V_COL") AS
select object_name from user_objects
where object_type='VIEW'
;

See the mistake? In the first definition, the closing semicolon was moved into the comment, rendering both view definitions useless, resulting in an "ORA-00933: SQL command not properly ended".

I could reproduce this if the views were created via Quest Toad (version 11.5) and Oracle SQL Developer (version 3.1) - but not when connecting directly via sqlplus (thin client version 11.2 or directly on the server). Did not have any other means of connecting to my databases to test - well, except for DBVisualizer 7.1.5, but that one just cuts off comments at object creation making a validation impossible.

Both a 10g (10.2.0.5) and an 11g (11.2.0.3) database showed the same behavior.


Wednesday, May 2, 2012

Count of targets not uploading exceeded the critical threshold

Whenever you get that alert in Enterprise Manager (this article is based on version 11.1.0.1) and click to see the details, you're left pretty much without useful information - it is not displayed which targets are not uploading data. Now you could click through all homepages of the targets of that host and see when the last upload happened in the All Metrics page, but that gets some serious work when having lots of targets on one machine.

Instead, fill in the host target's name in following SELECT, execute it in the EM repository database and you will see your problem target immediately:

select mt.target_name,
(select max(mmc.collection_timestamp)
from sysman.mgmt$metric_current mmc
where mmc.target_name=mt.target_name) last_collection
from sysman.mgmt$target mt
where host_name='[name of host target as displayed in EM]'
group by mt.target_name
order by last_collection;

On how to resolve the error - most of the times for databases a reconfiguration already helps (in the target list activate the bullet point next to the database target, click configure - test connection - next - submit). What will most definitely work is what also the Oracle Support will most likely suggest according to my experience: delete and re-add the target. Keep in mind that you will lose the historical data for it in the repository.

Also see following MOS documents: ID 1359328.1, ID 12612391.8