This is an old revision of the document!
Table of Contents
AKSW Cloud
Fabian has reminded us that we are only supposed to use the address range 192.168.35.96..127. Owners of the instances should change the IPv4 and IPv6 in %%(sql)/etc/network/interfaces%% and this wiki accordingly. Fabian: ich sehe nur auf der Switch, welche IPs wo liegen. Das sind auf Port 17: 75, 77, 78, 123, 124, 161, 201 Port 18: 110, 128, 139, 141, 143, 145, 146, 147, 162, 163, 180, 190 Port 16: 110, 138, 140, 191 (ohne die älteren IPs)
The AKSW cloud consists of 3 physical servers: #||
IP | Domain name | Description | |
139.18.2.226 2001:638:902:2010:0:168:35:100 | akswcc.informatik.uni-leipzig.de | Cloud Controller, Cluster Controller, Storage Controller | |
139.18.2.227 2001:638:902:2010:0:168:35:101 | akswnc1.informatik.uni-leipzig.de | Node Controller | |
139.18.2.228 2001:638:902:2010:0:168:35:102 | akswnc2.informatik.uni-leipzig.de | Node Controller | |
Add these to your ~~/.ssh/config (sql) Host akswcc Hostname 139.18.2.226 User uecadmin Host akswnc1 Hostname 139.18.2.227 User uecadmin Host akswnc2 Hostname 139.18.2.228 User uecadmin Host ubuntu-template Hostname 2001:638:902:2010:0:168:35:110 User akswadmin
Known Instances (sorted by IPv6): #||
IPv6 (IPv4) | Host | Host name | Owner | Description | new akswadmin pw | |
2001:638:902:2010:0:168:35:96 (139.18.2.79) | akswnc2 | aksw | Natanael | AKSW Website | ||
2001:638:902:2010:0:168:35:97 | akswnc2 | ldi | Seebi | Linked Data Infrastructure | ||
2001:638:902:2010:0:168:35:98 | akswnc2 | owdev | Philipp | OntoWiki Continuous Integration Server | ||
2001:638:902:2010:0:168:35:99 | akswnc2 | latc-legal | Philipp | LATC Legal Dataset Hosting | ||
100-103 reserved for host machines! | ||||||
2001:638:902:2010:0:168:35:104 | akswnc1 | wikipedia-oai-proxy | Claus | |||
2001:638:902:2010:0:168:35:105 | akswnc2 | Ivan | CubeViz Server | |||
2001:638:902:2010:0:168:35:107 | N/A | Mohamed | AKSWDISKTOP Server | |||
2001:638:902:2010:0:168:35:108 | akswcc | Conti-Interior Freebase | ||||
2001:638:902:2010:0:168:35:109 | akswcc | Suresh Pokharel | ||||
2001:638:902:2010:0:168:35:110 | all | Claus | Template Instance for cloning | |||
2001:638:902:2010:0:168:35:111 | akswcc | Claus | API Leipzig | |||
2001:638:902:2010:0:168:35:112 | akswcc | Axel | BioASQ | |||
2001:638:902:2010:0:168:35:113 | akswcc | Lorenz | ~AutoSparql | |||
2001:638:902:2010:0:168:35:114 | akswnc2 | Jens, Claus | ~GeoKnow | |||
2001:638:902:2010:0:168:35:115 | akswnc2 | Seebi | ~LEDS | |||
2001:638:902:2010:0:168:35:116 | akswnc1 | linkeddata-qa-production | Claus | lie3deiR | ||
2001:638:902:2010:0:168:35:117 | akswnc1 | Amrapali | Amrapali | |||
2001:638:902:2010:0:168:35:118 | akswnc2 | Jens | jenslehmann1 | |||
2001:638:902:2010:0:168:35:119 | akswncc | Konrad Hoeffner | konradhoeffner | |||
2001:638:902:2010:0:168:35:120 | akswnc2 | lgd-storage | Claus | |||
2001:638:902:2010:0:168:35:121 | akswnc3 | lgd-db | Claus | |||
2001:638:902:2010:0:168:35:122 | akswnc1 | service-monitor1 | Claus | Service Monitor | ||
2001:638:902:2010:0:168:35:123 | akswnc2 | relze | Claus | |||
2001:638:902:2010:0:168:35:124 | akswnc2 | cstadler | Claus | All my web frontends - PanLex, LGD, Sparqlify … | ||
2001:638:902:2010:0:168:35:125 | akswnc1 | tr-luther | Thomas | |||
2001:638:902:2010:0:168:35:126 | akswnc2 | tr-softwiki1 | Ivan | Prod-Server u.a. wiki.publicdata.eu | ||
2001:638:902:2010:0:168:35:128 | akswnc2 | seppl | Seppl | NIF Services for the 3.2.1 deliverable (demo.nlp2rdf.org) – NLP2RDF | shie0aiY | |
2001:638:902:2010:0:168:35:129 | akswnc1 | tr-publink | Thomas | Publink Data-Endpunkt der Neuen Deutschen Biographie | ||
2001:638:902:2010:0:168:35:130 | akswnc1 | tr-cpl-protected | Thomas | Protected Zone des Professorenkatalogs | ||
2001:638:902:2010:0:168:35:131 | akswnc2 | Sina Project | Saeedeh | |||
2001:638:902:2010:0:168:35:132 | akswcc | akswblogs | Edgard Marx | Aksw multiuser blog | ||
2001:638:902:2010:0:168:35:133 | akswnc2 | Ivan | CSV2RDF Server - 14.04 | |||
2001:638:902:2010:0:168:35:135 (139.18.2.130) | akswnc2 | LOD2-WP7-II | Seebi | LOD2 WP7 Hosting (wp7.lod2.eu) | zie6OPho | |
139.18.2.134 (was 2001:638:902:2010:0:168:35:136) | akswnc2 | Seebi | LOD2 Stack (stack.lod2.eu) | |||
2001:638:902:2010:0:168:35:137 | akswncc | Claus | Maven/Archiva | |||
139.18.2.152 | akswnc2 | Seebi | LOD2 FAO | AKSW LOD2 Publink 2012 | ||
2001:638:902:2010:0:168:35:138 | akswcc | Lorenz | ||||
2001:638:902:2010:0:168:35:139 | akswnc2 | Micha | LodStats | |||
2001:638:902:2010:0:168:35:140 | akswnc1 | PatrickWestphal | Patrick | Patrick VM | ||
2001:638:902:2010:0:168:35:141 | akswnc2 | Florian | SparqlCache | |||
2001:638:902:2010:0:168:35:142 | akswnc1 | Pfarrerbuch | Nate | Project on historic data of protestant pastors | ??? | |
2001:638:902:2010:0:168:35:143 | akswnc2 | ClausBenchmarking | Claus | temporary benchmarking server | ??? | |
2001:638:902:2010:0:168:35:144 | akswnc1 | mmoon project | Bettina | Mmoon data server and documentation | ??? | |
2001:638:902:2010:0:168:35:145 | akswnc2 | Claus | Benchmarks | |||
2001:638:902:2010:0:168:35:146 | akswnc2 | Claus | (Student) Demo Server | |||
2001:638:902:2010:0:168:35:147 | akswnc1 | AbbreviationBase | Martin | abbreviation base extraction server | ||
2001:638:902:2010:0:168:35:148 | akswnc1 | MichaelMartinSeminar | Michael | seminar host | ||
2001:638:902:2010:0:168:35:149 | akswnc1 | LOD2_final | Jens/Bert v Nuffelen | lod2 stack finalization | ||
2001:638:902:2010:0:168:35:150 - 155 | akswnc1 | Claus | LinkedData-QA-1 -5 | |||
outdated: 139.18.2.152, reassigned to lod2-fao | akswnc2 | dssn | Seebi | Distributed Social Semantic Networking | ??? | |
2001:638:902:2010:0:168:35:160 | akswnc1 | IvanErmilov | Ivan | used for FOX and other services hosting | ||
2001:638:902:2010:0:168:35:161 | akswnc1 | dataid | Ivan | dataid.dbpedia.org | ||
2001:638:902:2010:0:168:35:162 | akswnc2 | aled | Seebi | Active Learning of Edit Distances | ??? | |
2001:638:902:2010:0:168:35:163 | akswnc2 | Mukava | Timofey | Project on geo-locating music preferences using mobile devices | ??? | |
2001:638:902:2010:0:168:35:164 | akswnc2 | SakeProject | IvanErmilov | Web portal for the Sake project - project for complex stream events | ??? | |
2001:638:902:2010:0:168:35:165 | akswcc | IvanErmilov | Ivan | WackoWiki Delete Me in September 2015 | ||
2001:638:902:2010:0:168:35:166 | akswnc1 | IvanErmilov | Ivan | WackoWiki ziwd and liv legacy projects | ||
2001:638:902:2010:0:168:35:167 | akswnc2 | IvanErmilov | Ivan | idrac | ||
2001:638:902:2010:0:168:35:190 | akswnc2 | buildserver (ci.aksw.org) | Claus | |||
2001:638:902:2010:0:168:35:191 | akswcc | tdemo | Joerg | Telekom Suche Protoyp | ~WknXhgYdÜnGh | |
Instance Creation
- Log in to e.g. akswnc2
- (sql)ssh -X akswnc2
- -X does X11 forwarding, which means you can transfer some of the user interfaces
- LC_ALL=C virt-manager
- For some reason some Ubuntu update messed up locale settings and virt-manager won't start with the invalid locale config
- Clone the ubuntu-server-template - IMPORTENT: Adjust both the instance AND the storage name
- Log In to the freshly cloned instance
- (sql)ssh -X ubuntu-template
- Change hostname and IP
- cd cloud-tools
- ./ct-hostname-set.sh your-host-name
- ./ct-ip-set.sh 123 (only suffix needed)
- Update your .ssh/config file ;)
Important Notes
For each virtual instance there is an XML config files under (sql)/etc/libvirt/qemu. The virtual hard drives are located under (sql)/var/lib/libvirt/images/ !! Make sure that the bridge setting matches those of the other instances (br0 or br1)!!
Email Notification
Hi Instance Owner,
You can access your instance by adding the following entry to your ~/.ssh/config
Host {name-of-your-choice}
HostName 2001:638:902:2010:0:168:35:{your-ip} User akswadmin
Default password is: akswcloud, please change this ASAP.
You can then connect with ssh {name-of-your-choice}
Because of IPv6, you may need to do sudo apt-get install miredo sudo service miredo restart
Cheers, The AKSW cloud management team
Graphical Virt-manager
(sql) ssh -X uecadmin@139.18.2.226 -s virt-manager
If your ssh key has been added to akswcc, you can connect with: (sql)ssh uecadmin@akswcc.informatik.uni-leipzig.de When logged in, there is a README file containing further passwords.
The webadmin interface is available at: (sql) https://139.18.2.226:8443 https://akswcc.informatik.uni-leipzig.de:8443 You can apply for an account there, which then has to be approved by an administrator (usually Claus).
Basically all administrative task concerning the instances are performed on the akswcc.
Tutorials, Documentations and Guides
Instance Management via HybridFox
Once you applied for an account at https://akswcc.informatik.uni-leipzig.de:8443, you can manage instances via FireFox using the HybridFox plugin. Perform the following setup steps:
- install the latest 1) XPI file and restart Firefox
- go to Extras/Tools ⇒ HybridFox which should open a new tab
- click “Regions” and enter name “AKSW-Cloud” and enpoint URL “http://139.18.2.226:8773/services/Eucalyptus”
- in a seperate window, go to the 2) » credentials » show key
- click “Credentials” and fill the form using the values in the webadmin interface window: “Account Name” is your login name, “AWS Access Key” is the query ID, “AWS Secret Access Key” is the secret key
Creating Instances
Beforehand, please note that instances are transient. This means, as soon as they terminate they disappear (and with them, all data). However, it is possible to create virtual hard drives (called volumes) whose existence is independent of that of the instances. Volumes can then be attached to instances where they appear as ordinary devices (e.g. as /dev/sdb). Read here for way of achieving persistence in the 3)
For now, once you logged in as uecadmin to the cc, you will find scripts for running instances and connecting to them. There is also a script (taken from a tutorial) for creating the keypairs.
Change Resources (RAM, CPU, HDD) for Existing Instances
Eucalyptus Cloud itself comes with 5 profiles, that can be assigned user definde values for the amount of CPUs, RAM and Disk Space.
TODO I could't figure out the difference between m and c. Are there any special implications - or are these just names? TODO Make a table with resource limits.
m1.small m1.large c1.small c1.medium c1.xlarge
Change Resources (RAM, CPU, HDD) for Existing Instances with virt-manager
RAM: Open the existing instance and change the value under “Memory”. Reboot the instance DISK: Appending to an existing disk is very complicated. Adding another disk is relatively easy: -Open the existing instance → Add Hardware -Select “Storage” → forward -Change the value under “create disk image on the computer's hard drive” → forward → finish -The disk will be added to the instance. -To mount it on the computer follow these steps after point 6., default device will be /dev/sda: 4)
FAQ
- How can I set the range of public ip addresses?
(sql) vim /etc/eucalyptus/eucalyptus.local.conf Add the line: VNET_PUBLICIPS="139.18.2.230-139.18.2.233"
Virt-Manager
A simple way to create (persistent) virtual instances similar to the ones we have with Hyper-V can be achieved with the following instructions. Note that this solution is not meant to replace the cloud, as it lacks features such as automatically allocating instances on node-controllers with free resources. Instead, the following procedure has to be performed on every single computer that should host such virtual instances. (except for maybe virt-manager, which can probably be configured to be a front end for remote connections.)
(sql) sudo apt-get install virt-manager libvirt-bin qemu kvm
Add us to the kvm-group (sql) sudo adduser $USER kvm
Uncomment the lines in /etc/libvirt/qemu.conf (sql) # The user ID for QEMU processes run by the system instance user = "root" # The group ID for QEMU processes run by the system instance group = "root"
Restart the service (sql) sudo service qemu-kvm restart
Now you can do (sql) ssh -X <host> virt-manager
This will open a window, where you can configure connections and instances. If no connection yet exists, create one: (sql) File->Add Connection Hypervisor: QEMU/KVM Connection: Local (Note: The window we see is actually running on the host we ssh'd into and is forwarded to us). Autoconnect: Checked Connect
Done :)
Troubleshooting Virt-Manager
Interestingly, the above procedure has a success-rate of < 100% of practically identical systems: It worked on akswcc and akswnc2, but failed on akswnc1. * “Failed to spawn the configuration server (gconfd): Failed to contact configuration server; the most common cause is a missing or misconfigured D-Bus session bus daemon. See http://projects.gnome.org/gconf/ for information. (Details - 1: Failed to get connection to session: Error connecting: Connection refused)”
Reinstalling dbus and gconf2 and rebooting solved the problem at least for us. (sql) sudo apt-get install --reinstall dbus gconf2 sudo reboot
IPv6 Host Configuration
Make sure /etc/sysctl.conf contains the following lines: (sql) net.ipv6.conf.eth0.accept_ra = 0 net.ipv6.conf.eth0.autoconf = 0 net.ipv6.conf.eth1.accept_ra = 0 net.ipv6.conf.eth1.autoconf = 0 net.ipv6.conf.br0.accept_ra = 0 net.ipv6.conf.br0.autoconf = 0 Apply the changes using (sql)sysctl -p /etc/sysctl.conf
Example configuration for /etc/network/interfaces: (sql) auto eth0 iface eth0 inet static address <adjustme>139.18.2.228 netmask 255.255.255.0 network 139.18.2.0 broadcast 139.18.2.255 gateway 139.18.2.254 dns-nameservers 139.18.1.2 dns-search informatik.uni-leipzig.de # dns-* options are implemented by the resolvconf package, if installed auto eth1 iface eth1 inet manual # Configure the brigde for both IPv4 and IPv6 # Note the the brigde itself also needs an address auto br0 iface br0 inet static address <adjustme>192.168.35.99 netmask 255.255.254.0 bridge_ports eth1 bridge_fd 9 bridge_hello 2 bridge_maxage 12 bridge_stp off iface br0 inet6 static address <adjustme>2001:638:902:2010::168:35:99 netmask 64 gateway 2001:638:902:2010::1 bridge_ports eth1 bridge_fd 9 bridge_hello 2 bridge_maxage 12 bridge_stp off Apply the changes using (sql)/etc/init.d/networking restart
Route output should be (sql) Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface localnet * 255.255.255.0 U 0 0 0 eth0 192.168.122.0 * 255.255.255.0 U 0 0 0 virbr0 192.168.34.0 * 255.255.254.0 U 0 0 0 br0 default 139.18.2.254 0.0.0.0 UG 100 0 0 eth0
IPv6 Instance Configuration
Example configuration for /etc/network/interfaces on the virtual instance: (sql) auto lo iface lo inet loopback auto eth0 iface eth0 inet6 static address <adjustme>2001:638:902:2010::168:35:100 netmask 64 gateway 2001:638:902:2010::1 dns-nameservers 2001:638:902:1::10 Apply the changes using (sql)/etc/init.d/networking restart
Don't forget to edit /etc/resolv.conf for the nameserver: (sql) nameserver 2001:638:902:1::10
IPv6 and ssh'ing to the Instances
If ssh <user>@<ipv6> does not work, the problem might be that you are on an IPv4 net, but want to talk to IPv6 servers. In this case you need to tunnel your IPv6 packages through the IPv4 net, which works out-of-the-box after installing miredo: (sql) sudo apt-get install miredo
Exposing your cloud instance to the WWW
You can use HTTP reverse prxies to expose http endpoints like webinterfaces to the braod public. Lets assume you have a running cloud instance with the following services:
(sql) 192.168.35.123:8890 (Virtuoso) 192.168.35.123:8080 (Tomcat)
The goal is to expose them under a subdomain of aksw.org, for example
(sql) http://mysuperservice.aksw.org/virt http://mysuperservice.aksw.org/tomcat
You can expose them using a cascade of reverse proxies.
The first reverse proxy needs to be configured on prod0. This reverse proxy will do the name resolving and delegation to your server. Add therefore the an new file to sites-available with containing the following information:
<VirtualHost *> ServerName mysuperservice.aksw.org ServerAlias www.mysuperservice.aksw.org ServerAdmin webmaster@aksw.org ProxyRequests Off ProxyPreserveHost On # don't lose time with IP address lookups HostnameLookups Off # needed for named virtual hosts UseCanonicalName Off <Proxy *> Order allow,deny Allow from all </Proxy> ProxyPass / http://192.168.35.123/ retry=1 ProxyPassReverse / http://192.168.35.123/ <Location /> Order allow,deny Allow from all </Location> LogLevel warn CustomLog /var/log/apache2/mysuperservice.aksw.org/access.log combined ErrorLog /var/log/apache2/mysuperservice.aksw.org/error.log </VirtualHost> Restart the apache service. IMPORTANT: do not forget to create the directory for the logfiles. MORE IMPORTANT: do not mess around with this web server! The second step is enable the rewriting on your machine. Here you can mess around. Make sure that apache with mod_proxy and mod_proxy_html is installed and the mods are enabled. Add a new virtual host to your machine, again by adding an file into sites-available and sites-enabled. This file would contain in our case:
<VirtualHost *:80> ServerName mysuperservice.aksw.org #### For Virtuoso ProxyRequests off ProxyPass /virt/ http://192.168.35.123:8890/ ProxyPassReverse /virt/ http://192.168.35.123:8890/ ProxyHTMLURLMap http://192.168.35.123:8890 /virt <Location /virt/> ProxyPassReverse / # uncomment next line for version of mod_proxy_html => 3.1 # ProxyHTMLEnable On # use next line for version of mod_proxy_html < 3.1 SetOutputFilter proxy-html ProxyHTMLURLMap / /virt/ RequestHeader unset Accept-Encoding </Location> ####### For Tomcat ProxyRequests off ProxyPass /tomcat/ http://192.168.35.123:8080/ ProxyPassReverse /tomcat/ http://192.168.35.123:8080/ ProxyHTMLURLMap http://192.168.35.123:8080 /tomcat <Location /tomcat/> ProxyPassReverse / # uncomment next line for version of mod_proxy_html => 3.1 # ProxyHTMLEnable On # use next line for version of mod_proxy_html < 3.1 SetOutputFilter proxy-html ProxyHTMLURLMap / /tomcat/ RequestHeader unset Accept-Encoding </Location> </VirtualHost> Restart the apache service. Now your virtuoso is reachable under (sql) http://mysuperservice.aksw.org/virt and your Tomcat under (sql) http://mysuperservice.aksw.org/tomcat
Be advised that http://192.168.35.123/tomcat won't work because of of the virtual host settings on the cloud instance.