How to configure DataStax Enterprise

DataStax Enterprise

DataStax Enterprise is a NoSQL database platform that provides continuous availability, fast performance, strong security, and operational simplicity for the enterprise. Powered by Apache Cassandra, the exceptionally scalable and reliable NoSQL database, DataStax Enterprise tackles the complete spectrum of big data challenges that traditional databases and other NoSQL solutions can’t.

How to configure DataStax Enterpise

Requirements:

• Configure Windows Server (How to configure  Windows Server 2008 for DSE)
• Configure Linux (How to configure Linux for DSE)

Scheme DSE network

1.        Kerberos configuration Windows Server

1.1          Active Directory Users and Computers > Users > test_user (cassandra user)  and dse/linuxccm.cognet.local (cassandra server) > properties > tab account > Account options > set options:

·         Use Kerberos DES encryption

·         The account supports Kerberos AES 128 and 256 bit encryption

           (see images below)

Account options for “test_user” user (cassandra client)

Account options for "dse/linuxccm.cognet.local" user (cassandra server)

                             

1.2          Add group (Account Operators) for test_user (cassandra client) and dse/linuxccm.cognet.local (cassandra server) > properties > tab Member Of  

Add group for "dse/linuxccm.cognet.local"
user (cassandra server)

Add group for "test_user" user (cassandra client)

1.3         For dse/linuxccm.cognet.local (cassandra server) > properties > tab Delegation > set Trust this user for any services ( Kerberos only) 

Set trust for "dse/linuxccm.cognet.local" user (cassandra server)

Run ADSI Edit > Action > Connect > CN=cognet > CN=Computers > CN="Tester-PC"(cassandra client) > Properties > Tab Attribute Editor > msDS-SupportedEncryptionTypes >   set value:  for users 0x1B, for computers 0x1F.  (see images below)

Supported encryption  for "TESTER-PC"
 computer (windows client)

Supported encryption for "test_user" 

user (cassandra client)

1.4         And attribute: UserAccountControl > Add bit 0x200000 for enable DES key only

For computers 0x81000, for users linux 0x290200, for users windows 0x210200.

Attribute: UserAccountControl for
 "test_user" user (cassandra
 client)

Attribute: UserAccountControl for
"dse/linuxccm.cognet.local" user 
(cassandra server)

Attribute: UserAccountControl
 for 

"Tester-PC" computer (windows
 client)

1.5          Enable the following Group Policies to apply the DES encryption type to all computers that are running Windows 7(Tester-PC) or Windows Server 2008 R2 (cogserver01):

•  Run gpedit.msc (win button > type gpedit.msc) > Computer Configuration > Windows Settings > Security Settings > local policies > Security options >  Network security: Configure encryption types allowed for Kerberos >Tab: Local security settings set as image below (set local security options) enable all encryption type.

More information: Enable group policy

1.6          Create account mappings

a.      Start the Active Directory Users and Computers snap-in. Point to Programs, then Administrative tools, and then Active Directory Users and Computers.

b.      Start advanced features by clicking View, and then Advanced Features.

c.    Locate the account where you want to create mappings, and right-click on user to view Name Mappings. This example uses the account test_user.

d.     Click the Kerberos Names mappings tab.

e.    Add a principal from the foreign MIT (based on UNIX) realm. The example shown in Figure 10 below users test_user@COGNET.LOCAL (this user must create in Cassandra and Kerberos database)

For dse/linuxccm.cognet.local add kerberos principal: dse/linuxccm.cognet.local@COGNET.LOCAL (this user must create in Cassandra and Kerberos database)

1.7          Create keytab (on Domain Controller) for Linux

Run cmd > type:

C:> Ktpass -princ dse/linuxccm.cognet.local@COGNET.LOCAL -mapuser linuxccm -pass Cognitum2014 -crypto DES-CBC-MD5 -ptype KRB5_NT_PRINCIPAL -out c:/dse.keytab

More information: Ktpass settings

2.        Configuration Kerberos on Linux

2.1       DSE installation

2.1.1          Register and download  DSE (tarball version ) on Linux:

http://www.datastax.com/download/register/versions

2.1.2          Check the version of Java that is installed:

    $ java –version

Use the latest version of Oracle Java 6 or 7 on all nodes. If you need help installing Java, see Installing the Oracle JRE and JNA.

Check which installed Java you use and choose recommended version Java:

    $ sudo update-alternatives --config java

2.1.3          Unpack the distributions:

    $ tar -xzvf dse.tar.gz

2.1.4          Delete gz file:

    $ rm *.tar.gz

More information about installation tarball version: install tarball

2.2      Copy keytab from Domain Controller (C:/dse.keytab) to Linux (<dse_dir>/resources/dse/conf)  

2.2.1   Edit cassandra.yaml file (<install_location>/ resources/ cassandra/ conf).

 line 266,357,391

 set ip Linux for example 192.168.1.1

1)          line 266 - seeds: 127.0.0.1 for example set: - seeds: 192.168.1.1

2)          line 357 listen_address: localhost for example set: listen_address: 192.168.1.1

3)          line 391 rpc_address: localhost for examplet set: rpc_address:192.168.1.1

2.2.2   Edit cassandra-env.sh file (<install_location>/resources/cassandra/conf/cassandra-env.sh.)

comment out:

    JVM_OPTS="$JVM_OPTS -Denable-old-dse-state=true"

and:

    JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"

(found 13 lines from the bottom of the file)

If use Java 7 or later comment out:

    if [ "$JVM_VERSION" \> "1.7" ] ; then
JVM_OPTS="$JVM_OPTS -XX:+UseCondCardMark"
Fi

3.        Check the DSE

3.1          Run DSE (with parameter cassandra) for example:

    $. /<install_location>/bin/dse cassandra

3.2          Run putty on another computer (for example Client PC) and connect with Ubuntu

ip: linux ip (ip Linux Host)

user: user (user name in Linux)

password: password (password for user)

type:

$./<install_location>/bin/cqlsh <ip_Linux>

If you see the message in the image below, then DSE runs ok.

3.3         Download driver

https://github.com/datastax/csharp-driver/tree/release_1_0_0_dse

and

https://github.com/datastax/csharp-driver/tree/dse_3.1

3.4          Copy drivers to client computer (on Windows Client)

3.5          Start the Visual Studio 2010  snap-in or later version visual studio.

3.6          First open “release_1_0_0_dse” driver (<csharp-driver_dse>\Cassandra.MyTest.sln)

3.7          Edit line 

Cluster cluster = Cluster.Builder().AddContactPoint("192.168.0.116")   

 in Project Playground in Program class, change ip (192.168.0.116) to ip Linux[linuxccm] (192.168.1.1).

3.8          Right click on Playground project, then “set StartUp Project”

3.9          Build Solution (F6)

3.10       Start debugging (F5)

If ok (see image below),

Then edit cassandra.yaml file (<install_location>/resources/cassandra/conf).

line 85 comment out:

    authenticator: org.apache.cassandra.auth.AllowAllAuthenticator 

line 86 uncomment:

    #authenticator: org.apache.cassandra.auth.PasswordAuthenticator

line 100 comment out:

   authorizer: org.apache.cassandra.auth.AllowAllAuthorizer

line 101 uncomment:

   #authorizer: org.apache.cassandra.auth.CassandraAuthorizer

4.        Add Cassandra user:

Restart DSE:

$sudo killall java
$./<install_location>/bin/cqlsh  <ip_Ubuntu> -u cassandra -p cassandra

create user for example:

CREATE USER test WITH PASSWORD 'foo' SUPERUSER;
CREATE USER 'test_user@COGNET.LOCAL' WITH PASSWORD 'Cognitum2014' SUPERUSER;
CREATE USER 'dse/linuxccm.cognet.local@COGNET.LOCAL' WITH PASSWORD 'Cognitum2014' SUPERUSER;

4.1          Start the Visual Studio 2010(on Windows Client)  snap-in or later version visual studio.

4.2          Open “release_1_0_0_dse” driver (<csharp-driver_dse>\Cassandra.MyTest.sln)

4.3          Edit Playground/Program.cs line 44 from .WithAuthProvider(new DseAuthProvider()) to .WithAuthProvider(new PlainTextAuthProvider(“test”,”foo”))

4.4          Run Playground (If you see the message in the image below, then the DSE with password authentication runs ok)

or run debug-cql(on Linux Host):

·         Run debug-cql  that interaction would look like:

user@LinuxCCM:~/DSE/dse-3.2.4/resources/cassandra/bin$ ./debug-cql 192.168.1.1 9042
CQL binary protocol console 192.168.1.1@9042
Connecting...
>> STARTUP
-> AUTHENTICATE org.apache.cassandra.auth.PasswordAuthenticator
>> AUTHENTICATE username=test password=foo
-> org.apache.cassandra.transport.messages.AuthSuccess@3e0ebb

4.5          Check /etc/hosts (see image below)

5.        KDC Setup

$ sudo apt-get install krb5-kdc krb5-admin-server
$ sudo dpkg-reconfigure krb5-kdc

The package installation process will step through defining the basic Kerberos configuration parameters. Recommended settings are:

·         disable Kerberos 4 compatibility mode

·         do not run krb524d (daemon to convert Kerberos tickets between versions)

·         defaults for the other settings are acceptable

Here is an example configuration file(/etc/krb5kdc/kdc.conf):

Supported_enctypes we are using only des-cbc-md5:normal

Edit file /etc/krb5kdc/kadm5.acl:

Create the Kerberos database with the following command:

$sudo krb5_newrealm

6.        Realm Administration: kadmin

To start the kadmin utility, issue the following command:

$sudo kadmin.local

Add a user: 

kadmin: addprinc test_user

Add a service: 

kadmin: addprinc dse/linuxccm.cognet.local

7.        Client configuration

Edit /etc/krb5.conf (as in the image below)

8.        Testing kerberos configuration

Run DSE/Cassandra

$<install_location>sudo bin/dse Cassandra

To test the operation of Kerberos, request a Ticket-Granting Ticket (TGT) with the kinit command

$ kinit –p test_user@COGNET.LOCAL
Password for test_user@COGNET.LOCAL:

Use the klist command to verify the TGT is valid:

$ klist
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: test_user@COGNET.LOCAL
Valid starting     Expires            Service principal
02/11/14 08:07:14  02/11/14 18:09:43  krbtgt/COGNET.LOCAL@COGNET.LOCAL
renew until 02/12/14 08:07:14

More information about Kerberos configuration: Kerberos

9.        DSE Configuration 

9.1          Edit dse.yaml (<install_location>/resources/dse/conf) (as in the example below).

keytab: resources/dse/conf/dse.keytab
service_principal: dse/_HOST@COGNET.LOCAL
http_principal: HTTP/_HOST@COGNET.LOCAL
qop: auth

9.2          Edit cassandra.yaml file (<install_location>/resources/cassandra/conf).

line 86 comment out:

authenticator: org.apache.cassandra.auth.PasswordAuthenticator

       
         line 88 uncomment:

#authenticator: com.datastax.bdp.cassandra.auth.KerberosAuthenticator

9.3          Edit Playground/Program.cs line 44 from  .WithAuthProvider(new PlainTextAuthProvider("test","foo")) to  .WithAuthProvider(new DseAuthProvider())

10.   Testing DSE with Kerberos authentication

10.1       To work around debug-cql, you can just disable JMX option in cassandra-env   ((<install_location>/resources/cassandra/conf/Cassandra-env.sh) by commenting out this line:

JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"

(found 13 lines from the bottom of the file)

10.2       Run DSE/Cassandra

$<install_location>sudo bin/dse Cassandra

If you want more information about debugging run dse with parametr JVM_OPTS="-Dsun.security.krb5.debug=true"(debugging info) and –f (foreground):

$<install_location>sudo JVM_OPTS="-Dsun.security.krb5.debug=true" bin/dse cassandra –f

10.3       Run debug-cql

a)      Set ticket for user:

$ kinit –p test_user@COGNET.LOCAL

b)      Check generated ticked:

$ klist

Look at Testing kerberos configuration

c)       Run debug-cql  that interaction would look like:

$<install_location>/resources/cassandra/bin$ JVM_OPTS="-Dkerberos.enabled=true" ./debug-cql linuxccm.cognet.local 9042 CQL binary protocol console linuxccm.cognet.local@9042
Connecting...
>> STARTUP
-> AUTHENTICATE com.datastax.bdp.cassandra.auth.KerberosAuthenticator
>> AUTHENTICATE
-> org.apache.cassandra.transport.messages.AuthSuccess@1b0bdc8

If you want more information about debugging run debuq-cql with parametr JVM_OPTS="-Dsun.security.krb5.debug=true”(debugging info):

$<install_location>/resources/cassandra$ JVM_OPTS="-Dsun.security.krb5.debug=true -Dkerberos.enabled=true" bin/debug-cql linuxccm.cognet.local 9042

11.   Troubleshooting

Kerberos is fairly fault-tolerant, if the requisite services are in place. If Kerberos authentication fails, check the following:

A.      The user has a valid ticket (use klist).

B.      “Authentication requires both 'username' and 'password'” check file cassandra.conf

C.      “Unable to obtain Princpal Name for authentication” check files: dse.keytab, /etc/krb5.conf, /etc/krb5kdc/kdc.conf and list_principals($sudo kadmin.local, then list_principals) if principal doesn’t exists, create account ($sudo kadmin.local,then add_princ name), principal must have the same account in cassnadra database and AD users.

D.      “Unable to obtain password from user”

a)      Check the contents of the keytab with: klist -e -t -k /path/to/keytab

b)      Use the kinit utility to verify that Kerberos is set up properly and that your principal and keytab are valid:  kinit -k -t keytab-file account-name

c)       Check supported encryption in settings on domain controller(ADSI edi, gpedit.msc and Active Directory Users and Computers)

d)      Restart all machine (important is restart Domain controller(cogserver01))

e)      Then generate again keytab:

Ktpass -princ dse/linuxccm.cognet.local@COGNET.LOCAL -mapuser linuxccm -pass Cognitum2014 -crypto DES-CBC-MD5 -ptype KRB5_NT_PRINCIPAL -out c:/dse.keytab

f)       Copy keytab to Linux host

g)      Edit dse.yaml file (authentication doesn't run correctly change keytab path to direct path).

E.      “kinit: KDC has no support for encryption type while getting initial credentials”

kinit fails with complaint about encryption type
To fix, edit /etc/krb5.conf file, and in the [libdefaults] section add

allow_weak_crypto = true

or reset password for test_user

F.      “kinit: Clock skew too great while getting initial credentials”

The clocks of the KDC and local machine aren’t synchronized. Check date and format of date.

$date
Wed Nov  2 15:52:07 MYT 2011
$ ntpdate -u 192.168.1.200
2 Nov 15:39:37 ntpdate[7948]: step time server 192.168.1.200 offset -751.194050 sec

More information: clock-skew-too-great-while-getting-initial-credentials

G.     Basic network connectivity is available (use ping and nslookup).

H.     Forward DNS hostname lookup succeeds on both the KDC and the local machine.

I.        Reverse DNS lookup succeeds on both the KDC and local machine, or dns is set to false in krb5.conf

J.       The clocks of the KDC and local machine are synchronized.

K.     The file permissions for various critical files such as /etc/krb5.keytab are correct and accessible by the user or service in question.sudo -u <username> can be helpful for this.

Following the KDC logs while attempting an operation can also be very informative. The following lines in /etc/krb5kdc/kdc.conf will enable KDC logging:

[logging]
       kdc = FILE:/var/log/krb5kdc.log

If you had to edit kdc.conf to enable logging, restart the KDC to apply the changes:

$ sudo service krb5-kdc restart

Once logging is enabled, you can follow the log while attempting an operation:

$ sudo tail -f /var/log/krb5kdc.log

More information about Kerberos: Kerberos and setup kerberos

---

Cognitum is a partner of DataStax, a major Cassandra vendor that provides worldwide training for Cassandra and Enterprise level appliances: DataStax Enterprise combining Cassandra, Hadoop, Hive, Solr into single solution. 
Cognitum provides IT solutions in the area of Cloud, BigData and Semantic Technology for customers both in Poland and abroad.