EPICS Archiver Appliance Installation Using an Install Script Log[Debian and CentOS Linux]

1 Archiver Appliance Installation

Archiver appliance[1] is download here.

user@user:~/Downloads# ls
archappl_v0.0.1_SNAPSHOT_23-September-2015T07-41-14.tar.gz

1.1 License

The EPICS archiver appliance is distributed under the terms of the EPICS license.

1.2 System requirements

A recent version of Linux

Definitely 64 bit Linux for production systems. If using RedHat, we should aim for RedHat 6.1

Sun Java JDK 1.8

Definitely the 64 bit version for production systems. We need the JDK, not the JRE.
A recent version of Tomcat 7.x
Preferably apache-tomcat-7.0.61 or later.

A recent version of Tomcat 7.x

Preferably apache-tomcat-7.0.61 or later.

Firefox or Chrome

The management UI works best with a recent version of Firefox or Chrome.
A recent version of MySQL mysql-5.1 or later
If persisting configuration to a database.

A recent version of MySQL mysql-5.1 or later

If persisting configuration to a database.

1.3 System Requirements Installation

JDK 1.8 installation

Sun Java JDK 1.8 is download here.
Here is the quick download log for all Linux user. In my case JDK 1.8 download the /opt directory.

root@user:/opt# wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2
Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u66-b17/jdk-8u66-linux-x64.tar.gz"

Untar the downloaded file.

root@user:/opt# tar -xzf jdk-8u66-linux-x64.tar.gz
root@user:/opt# ls
jdk1.8.0_66

Here is the installation log for Debian Linux user.

root@user:/opt# update-alternatives --install /usr/bin/java java /opt/jdk1.8.0_66/bin/java 1041
root@user:/opt# update-alternatives --install /usr/bin/javac javac /opt/jdk1.8.0_66/bin/javac 1041
root@user:/opt# update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).
Selection                                       Path                                    Priority       Status
------------------------------------------------------------
* 0     /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java   1071       auto mode
1     /opt/jdk1.8.0_66/bin/java                                               1041       manual mode
2     /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java     1071       manual mode
Press enter to keep the current choice[*], or type selection number: 1
update-alternatives: using /opt/jdk1.8.0_66/bin/java to provide /usr/bin/java (java) in manual mode
root@user:/opt# java -version
java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)

If you are using CentOS, here is the installation log.

[root@user opt]# alternatives --install /usr/bin/java java /opt/jdk1.8.0_66/bin/java 2
[root@user opt]# alternatives --config java
There are 4 programs which provide ’java’.
Selection           Command
-----------------------------------------------
*+ 1     /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java
2     /usr/lib/jvm/jre-1.6.0-openjdk.x86_64/bin/java
3     /usr/lib/jvm/jre-1.5.0-gcj/bin/java
4     /opt/jdk1.8.0_66/bin/java
Enter to keep the current selection[+], or type selection number: 4
[root@user opt]# alternatives --install /usr/bin/jar jar /opt/jdk1.8.0_66/bin/jar 2
[root@user opt]# alternatives --install /usr/bin/javac javac /opt/jdk1.8.0_66/bin/javac 2
[root@user opt]# alternatives --set jar /opt/jdk1.8.0_66/bin/jar
[root@user opt]# alternatives --set javac /opt/jdk1.8.0_66/bin/javac
[root@user opt]# java -version
java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)

Tomcat 7.x Download

Tomcat 7.x is download here.

user@user:~/Downloads# ls
apache-tomcat-7.0.65.tar.gz
archappl_v0.0.1_SNAPSHOT_23-September-2015T07-41-14.tar.gz

Chrome installation

Chrome is download here.
Here is the chrome installation log for Debian Linux user.

user@user:~/home/user/Downloads# ls
apache-tomcat-7.0.65.tar.gz
archappl_v0.0.1_SNAPSHOT_23-September-2015T07-41-14.tar.gz
google-chrome-stable_current_amd64.deb
user@user:~/Downloads$ su
Password:
root@user:/home/user/Downloads# dpkg -i google-chrome-stable_current_amd64.deb
root@user:/home/user/Downloads# aptitude install google-chrome-stable

MySQL installation and setting

Create MySQL user and table is needed for the archiver appliance installation.

Here is the quick installation log for Debian Linux user. Other Linux users install the same name package.

Package : mysql-server

root@user:~# aptitude install mysql-server

MySQL connector download here, and untar the download file.

user@user:~/home/user/Downloads# ls
apache-tomcat-7.0.65.tar.gz
archappl_v0.0.1_SNAPSHOT_23-September-2015T07-41-14.tar.gz
google-chrome-stable_current_amd64.deb
mysql-connector-java-5.1.38.tar.gz
user@user:~/home/user/Downloads# tar -xzf mysql-connector-java-5.1.38.tar.gz
user@user:~/home/user/Downloads# ls
apache-tomcat-7.0.65.tar.gz
archappl_v0.0.1_SNAPSHOT_23-September-2015T07-41-14.tar.gz
google-chrome-stable_current_amd64.deb mysql-connector-java-5.1.38.tar.gz
mysql-connector-java-5.1.38

Here is the create MySQL user and table log for Debian Linux user.

user@user:~$ mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 66
Server version: 5.5.46-0+deb8u1 (Debian)
Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type ’help;’ or ’\h’ for help. Type ’\c’ to clear the current input statement.
mysql> create user username@localhost identified by ’password’;
Query OK, 0 rows affected (0.00 sec)
mysql> create database databasename;
Query OK, 0 rows affected (0.00 sec)
mysql> grant all on databasename.* to username@localhost;
Query OK, 0 rows affected (0.00 sec)
mysql> exit
Bye
user@user:~$ mysql -u username -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 72
Server version: 5.5.46-0+deb8u1 (Debian)
Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type ’help;’ or ’\h’ for help. Type ’\c’ to clear the current input statement.
mysql> show databases;
+-------------------------+
| Database                              |
+-------------------------+
| information_schema          |
| databasename                     |
+-------------------------+
2 rows in set (0.00 sec)
mysql> use databasename;
Database changed
mysql> exit;
bye

If you are using CentOS, here is the create MySQL user and table log.

[root@user ~]# service mysqld start
Initializing MySQL database: Installing MySQL system tables...
OK
Filling help tables...
OK
To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system
PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:
/usr/bin/mysqladmin -u root password ’new-password’
/usr/bin/mysqladmin -u root -h node2 password ’new-password’
Alternatively you can run:
/usr/bin/mysql_secure_installation
which will also give you the option of removing the test
databases and anonymous user created by default. This is
strongly recommended for production servers.
See the manual for more instructions.
You can start the MySQL daemon with:
cd /usr ; /usr/bin/mysqld_safe &
You can test the MySQL daemon with mysql-test-run.pl
cd /usr/mysql-test ; perl mysql-test-run.pl
Please report any problems with the /usr/bin/mysqlbug script!
[ OK ]
Starting mysqld:
[ OK ]
[root@user ~]# /usr/bin/mysqladmin -u root password ’password’
[root@user ~]# mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.1.73 Source distribution
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type ’help;’ or ’\h’ for help. Type ’\c’ to clear the current input statement.
mysql> create user username@localhost identified by ’password’;
Query OK, 0 rows affected (0.00 sec)
mysql> create database databasename;
Query OK, 1 row affected (0.00 sec)
mysql> grant all on databasename.* to username@localhost;
mysql> show databases;
+-------------------------+
| Database                              |
+-------------------------+
| information_schema          |
| mysql                                    |
| databasename                     |
+-------------------------+
4 rows in set (0.00 sec)

1.5 Archiver Appliance Installation Using an Install Script

Install script accommodates installations with a ”standard” set of parameters and installs the EPICS archiver appliance on one machine.

If you want to simply test the system and quickly get going, please see the Quickstart section.

1

Make sure you have a recent version of Sun Java 1.8 from Oracle by running java -version.

You should see something like so

user@user:~$ java -version
java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)

2

The downloaded installation package to a Linux machine into a brand new folder. This should give you a tar.gz file like archappl vx.x.x.tar.gz. Also, Tomcat package into a brand new folder.

user@user:~$ mkdir archappl
user@user:~$ cd Downloads
user@user:~/Downloads$ mv archappl_v0.0.1_SNAPSHOT_23-September-2015T07-41-14.tar.gz apache-tomcat-7.0.65.tar.gz ./../archappl
user@user:~/Downloads$ cd ../archappl
user@user:~/archappl$ ls
apache-tomcat-7.0.65.tar.gz     archappl_v0.0.1_SNAPSHOT_23-September-2015T07-41-14.tar.gz

3

Untar the archappl vx.x.x.tar.gz. This should untar into 4 WAR files and a bash script

like so

user@user:~/archappl$ tar -xzf archappl_v0.0.1_SNAPSHOT_23-September-2015T07-41-14.tar.gz
user@user:~/archappl$ ls
Apache_2.0_License.txt
install_scripts RELEASE_NOTES
apache-tomcat-7.0.65.tar.gz
LICENSE
retrieval.war
archappl_v0.0.1_SNAPSHOT_23-September-2015T07-41-14.tar.gz mgmt.war
sample_site_specific_content
engine.war
NOTICE
etl.war
quickstart.sh

4

Set JAVA HOME to point to a 1.8 JDK

user@user:~/archappl$ export JAVA_HOME=/opt/jdk1.8.0.66

5

Run the install script like so

user@user:~/archappl$ cd install_scripts
user@user:~archappl/install_scripts$ ./single_machine_install.sh

This should start the install script.

6

Soon you can see the following window.

Pick a folder for create the Tomcat instance      

It is recommended that you select an empty folder. In my case, create a archiver appliance
folder and pick this folder.

7

Soon you can see the following window.

Pick a Tomcat distribution (tar.gz) file

 Select a Tomcat distribution (tar.gz) file.

8

Soon you can see the following window.

Pick a mysql client jar

 Select a mysql client jar file.

9

Soon you can see the following window.

Question about ARCHAPPL APPLIANCES environment variable

 In my case, select the YES button. The appliance.xml file is automatically created in the selected installation folder. And it is automatically export ARCHAPPL APPLIANCES. Of course, after the installation you can modify this file. If you want generate appliance.xml, following paragraph is the way for create appliance.xml and export appliance.xml.
The appliances.xml is a file that lists all the appliances in a cluster of archiver appliance. While it is not necessary to point to the same physical file, the contents are expected to be identical across all appliances in the cluster. The details of the file are outlined in the ConfigService javadoc. A sample appliances.xml with two appliances looks like

<appliances>
<appliance>
<identity>appliance0</identity>
<cluster_inetport>machinename:16670</cluster_inetport>
<mgmt_url>http://machinename:17665/mgmt/bpl</mgmt_url>
<engine_url>http://machinename:17666/engine/bpl</engine_url>
<etl_url>http://machinename:17667/etl/bpl</etl_url>
<retrieval_url>http://machinename:17668/retrieval/bpl</retrieval_url>
<data_retrieval_url>http://machinename:17668/retrieval</data_retrieval_url>
</appliance>
</appliances>
appliances.xml (END)

The archiver appliance looks at the environment variable ARCHAPPL APPLIANCES for the location of the appliances.xml file. Use an export statement like so

export ARCHAPPL_APPLIANCES=/location/appliances.xml

10

Soon you can see the following window.

MySQL information input window

 Enter a user name, password, databasename you set in MySQL. If not set, check the MySQL installation and setting section.

11

Soon you can see the following window.

Question about MySQL database table

 The archiver appliance ships with DDL, for MySQL, this is a file called archappl mysql.sql that is located in /home/user/archappl/install scripts/archappl mysql.sql. Execute this script in you newly created schema. We select the YES button.
There should be at least these tables

• PVTypeInfo - This table stores the archiving parameters for the PVs
• PVAliases - This table stores EPICS alias mappings
• ExternalDataServers - This table stores information about external data servers.
• ArchivePVRequests - This table stores archive requests that are still pending.

mysql> show tables;
+-----------------------+
| Tables_in_databasename|
+-----------------------+
| ArchivePVRequests           |
| ExternalDataServers        |
| PVAliases                          |
| PVTypeInfo                       |
+-----------------------+
4 rows in set (0.00 sec)

12

Soon you can see the following window.

Question about policies.py file

 We did not create the policies.py file. We choose the No button. For more mention is back to the policies.py file.

13

Soon you can see the following window.

Done with the installation window

Congratulations. Now just add a few step to complete installation.

14

Setting up Apache Commons Daemon

Editing the /apache-tomcat-7-0-65/conf/server.xml file to change the ports to better suit your installation

• By default, the connector port for the HTTP connector is set to 8080. Change this to the port used by the mgmt webapp for this appliance, in this example, 17665.

<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" />

 <Connector port="17665" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" />

•  Remove/comment out the sections for the AJP connector.

<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

 <!--Connector port="8009" protocol="AJP/1.3" redirectPort="8443" /-->

•  At the end, there should be two ports active in the /apache-tomcat-7-0-65/conf/server.xml file, one for the HTTP connector and the other for the SHUTDOWN command.

<Server port="8005" shutdown="SHUTDOWN">

You can ignore following paragraph about log4j.properties. The log4j.properties file is automatically created in apache-tomcat-7-0-65/lib folder. However, if you want a personal log settings, look in the following paragraphs.

Setting the appropriate log4j configuration level by creating/editing the apache-tomcat-7-0-65/lib/log4j.properties. Here’s a sample that logs exceptions and errors with one exception - log messages logged to the config namespace are logged at INFO level.

# Set root logger level and its only appender to A1.
log4j.rootLogger=ERROR, A1
log4j.logger.config.org.epics.archiverappliance=INFO
log4j.logger.org.apache.http=ERROR
# A1 is set to be a DailyRollingFileAppender
log4j.appender.A1=org.apache.log4j.DailyRollingFileAppender
log4j.appender.A1.File=arch.log
log4j.appender.A1.DatePattern=’.’yyyy-MM-dd
# A1 uses PatternLayout.
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n

15

Setting up sampleStartup.sh
Editing the /archiver appliance/sampleStartup.sh

• Setting up EPICS environment.

Default

source /opt/local/setEPICSEnv.sh

EPICS environment position.

source /home/user/epics/R3.14.12.5/setEpicsEnv.sh

• Setting up storage. 

Default

export ARCHAPPL_SHORT_TERM_FOLDER=/arch/sts/ArchiverStore
export ARCHAPPL_MEDIUM_TERM_FOLDER=/arch/mts/ArchiverStore
export ARCHAPPL_LONG_TERM_FOLDER=/arch/lts/ArchiverStore

Storage environment variables example

export ARCHAPPL_SHORT_TERM_FOLDER=/home/user/archiver_appliance/arch/STS
export ARCHAPPL_MEDIUM_TERM_FOLDER=/home/user/archiver_appliance/arch/MTS
export ARCHAPPL_LONG_TERM_FOLDER=/home/user/archiver_appliance/arch/LTS

 Congratulations. Archiver appliance installation is now complete.

16

Here is how to use the script.

Usage: ./sampleStartup.sh {start|stop|restart}

Run the archive appliance like so

user@user:~/archiver_appliance$ ./sampleStartup.sh start
Starting tomcat at location /home/user/archiver_appliance/mgmt
Using 64 bit versions of libraries
~/archiver_appliance/mgmt/logs ~/archiver_appliance
~/archiver_appliance
Starting tomcat at location /home/user/archiver_appliance/engine
Using 64 bit versions of libraries
~/archiver_appliance/engine/logs ~/archiver_appliance
~/archiver_appliance
Starting tomcat at location /home/user/archiver_appliance/etl
Using 64 bit versions of libraries
~/archiver_appliance/etl/logs ~/archiver_appliance
~/archiver_appliance
Starting tomcat at location /home/user/archiver_appliance/retrieval
Using 64 bit versions of libraries
~/archiver_appliance/retrieval/logs ~/archiver_appliance
~/archiver_appliance

Bibliography

[1] The EPICS Archiver Appliance : “archiver appliance”http://slacmshankar.github.io/epicsarchiver_docs/index.html (2016.01.06)
[2] Sang-il Lee(2014), “Archiver Appliance 구성”, RISP control group technical document