Concatenating lines in ldapsearch results

Many of us have had reasons to migrate Oracle Application Server (specifically, Portal) environments from one server or group of servers to another. This is often the case when hardware upgrades are needed and the whole environment must be moved to another set of hosts. Recently, I was helping move an Oracle Portal (10.1.2.0.2) environment … Continue reading “Concatenating lines in ldapsearch results”

Many of us have had reasons to migrate Oracle Application Server (specifically, Portal) environments from one server or group of servers to another. This is often the case when hardware upgrades are needed and the whole environment must be moved to another set of hosts.

Recently, I was helping move an Oracle Portal (10.1.2.0.2) environment from one host to another. This was due to a company spin off, so the “sticky” part of this move was that the domain name and resulting realm changed (more on that in a minute).

First, if you’ve had to perform this task, you should have already identified Metalink Note 251776.1 which describes the process necessary for moving users and groups from one OracleAS Infrastructure to another. The note’s step 3 mentions that the LDIF file must be edited to replace all references to the old realm with the new realm in the target system. However, this can prove difficult if you do actually have to change the realm name because of the way that ldapsearch produces output. The LDIF standard specifies that lines can be continued on the following line if a space is the first character on the line. The corresponding ldapadd command can properly import lines that are broken into multiple lines, but the standard search and replace tools (in notepad, vi or any other standard text editor) can’t find the occurrences properly to replace them. So, some entries are able to be replaced easily like this one (assume we need to replace “dc=dannorris,dc=local” with “dc=newcorp,dc=com“): 

dn: cn=dba,cn=portal.080827.121025.144000000,cn=groups,dc=dannorris,dc=local
uniquemember: cn=portal,cn=users,dc=dannorris,dc=local
uniquemember: cn=orcladmin,cn=users,dc=dannorris,dc=local
cn: DBA
displayname: DBA
description: portal.080827.121025.144000000 - Database Administrators
owner: cn=portal,cn=users,dc=dannorris,dc=local
owner: cn=orcladmin,cn=users,dc=dannorris,dc=local
objectclass: top
objectclass: groupofUniqueNames
objectclass: orclGroup
objectclass: orclPrivilegeGroup

The problem comes when you encounter a group name that causes one or more of the lines to be wrapped to the next line (ldapsearch has a hard-coded line length), like this one (note that the first and eighth lines aren’t wrapped because of your screen–that’s actually how ldapsearch outputs them):

dn: cn=DANCO_PROCUREMENT_LIMITED,cn=portal.080827.121025.144000000,cn=groups,d
 c=dannorris,dc=local
objectclass: orclGroup
objectclass: groupOfUniqueNames
objectclass: top
owner: cn=danco_admin,cn=users, dc=dannorris,dc=local
uniquemember: cn=danco_admin,cn=users,dc=dannorris,dc=local
uniquemember: cn=dba,cn=portal.080827.121025.144000000,cn=groups,dc=dannorris,
 dc=local
orclisvisible: true
displayname: DANCO_PROCUREMENT_LIMITED
cn: DANCO_PROCUREMENT_LIMITED

The standard search and replace function in the text editor won’t be able to interpret the value of the dn: attribute above as somethin that needs to be replaced because the search string would be “dc=dannorris,dc=local” and the above is “d\n c=dannorris,dc=local” instead. Similarly, the uniquemember: value is also broken across lines.

I found it useful to have a short utility script (in Perl, because that’s what I know and it’s always available with recent Oracle installations) that can concatenate the broken lines into a single line that can be consumed by search and replace more easily. The script also removes the authpassword: attributes as it processes since those can’t be loaded with ldapadd (as mentioned in the Metalink note). Here’s the script source (I call it concat.pl):

$cnt=0;
while ($line = <>) {
  chomp($line);
  if ( $line =~ /^authpassword/ ) {
    next;
  } elsif ( $line =~ /^\S+/) { ### this line is a "normal" or starting line
    $results[$cnt++] = $line;
  } elsif ( $line =~ /^$/ ) {  ### this line is blank
    $results[$cnt++] = "";
  } elsif ( $line =~ /^ \S+/ ) {  ### this line is a continuation
    $results[$cnt-1] = $results[$cnt-1] . substr($line,1);
  }
}

for $i (0 .. $cnt) {
  print "$results[$i]\n";
}

The script essentially creates a simple array of output and then the for loop at the end spits out the resulting array to STDOUT. There are a few simple regular expressions used and the \S character class from Perl to make things easy. To run this script, you need to invoke perl directly (I wrote this for Windows, so that’s why there’s no #! at the start). I run it like this on Windows:

%ORACLE_HOME%\perl\5.6.1\bin\MSWin32-x86\perl concat.pl < input.ldif > output.ldif

Then just review the output to see that it looks like you expect. Once satisfied, you can perform the search and replace knowing that it should match all the occurrences, not just the ones that happened to fit on a single line.

8 thoughts on “Concatenating lines in ldapsearch results”

  1. Very handy Dan. It hadn't actually occurred to me that you could always use Perl from the Oracle home before – I do often find myself having to do some sort of “munging” across multiple lines and Perl is probably the easiest tool for the job.

  3. So usefull !!!

    I got this issue with dipassistant boostrap (line breaks) and incorrect changetype attribute

    Thanks Dan !

  5. I was having a problem with this script. It doesn't handle the situation where the continuation line contains spaces. Here's a better version:

    $cnt=0;
    while ($line = <>) {
    chomp($line);
    if ( $line =~ /^authpassword/ ) {
    next;
    } elsif ( $line =~ /^S+/) { ### this line is a “normal” or starting line
    $results[$cnt++] = $line;
    } elsif ( $line =~ /^$/ ) { ### this line is blank
    $results[$cnt++] = “”;
    } elsif ( $line =~ /^s+(S.*)/ ) { ### this line is a continuation
    $results[$cnt-1] = $results[$cnt-1] . $1;
    }
    }

    for $i (0 .. $cnt) {
    print “$results[$i]n”;
    }

  6. I was having a problem with this script. It doesn't handle the situation where the continuation line contains spaces. Here's a better version:

    $cnt=0;
    while ($line = <>) {
    chomp($line);
    if ( $line =~ /^authpassword/ ) {
    next;
    } elsif ( $line =~ /^S+/) { ### this line is a “normal” or starting line
    $results[$cnt++] = $line;
    } elsif ( $line =~ /^$/ ) { ### this line is blank
    $results[$cnt++] = “”;
    } elsif ( $line =~ /^s+(S.*)/ ) { ### this line is a continuation
    $results[$cnt-1] = $results[$cnt-1] . $1;
    }
    }

    for $i (0 .. $cnt) {
    print “$results[$i]n”;
    }

  7. Many thanks for this post. There’s a little problem with this solution. If the ldif is very big the array could collapse the memory. In my case the ldif allocates 70 gb os disk space, so the script with the array crash.

    In this case I would prefer this script:

    #!/usr/bin/perl

    while ($line = <>) {
    chomp($line);
    if ( $line =~ /^authpassword/ ) {
    next;
    } elsif ( $line =~ /^S+/) { ### this line is a “normal” or starting line
    print “$complete_linen”;
    $complete_line=$line;
    } elsif ( $line =~ /^$/ ) { ### this line is blank
    print “$complete_linen”;
    $complete_line=””;
    } elsif ( $line =~ /^s+(S.*)/ ) { ### this line is a continuation
    $line =~ s/^s+//;
    $complete_line=$complete_line . $line;
    }
    }

    This script can be run in a pipe, like this:

    # gzcat backup_ldif.gz | ./concat_ldif.pl

  8. Pingback: David Fernandez » LDAPSearchWrapLines

Comments are closed.