source: README @ 1365:7cef08972437

Revision 1365:7cef08972437, 22.0 KB checked in by Babis Nikolaou <charnik@…>, 3 years ago (diff)

updated README file on how to run a specific test

Line 
1                        Strabon
2                      -----------
3
4Introduction
5============
6Strabon is a fully implemented semantic geospatial database system that can be
7used to store linked geospatial data expressed in RDF and query them using an
8extension of SPARQL. Strabon supports spatial selections, spatial joins, a rich
9set of spatial functions similar to those offered by geospatial relational
10database systems, support for multiple Coordinate Reference Systems and widely
11used serializations for geometric objects such as WKT and GML. Strabon is built
12on top of the well-known RDF store Sesame and extends Sesame’s components to be
13able to manage thematic and spatial data that are stored in PostGIS.
14
15The development of Strabon started in the context of European FP7 project
16SemsorGrid4Env (Semantic Sensor Grids for Rapid Application Development for
17Environmental Management) [http://www.semsorgrid4env.eu/]. Starting September
182011, Strabon is being utilized and extended with new functionalities in the
19FP7 project TELEIOS (Virtual Observatory Infrastructure for Earth Observation
20Data) [http://www.earthobservatory.eu/] which our group leads.
21
22The query language of Strabon is called stSPARQL. stSPARQL can be used to query
23data represented in an extension of RDF called stRDF. stRDF and stSPARQL have
24been designed for representing and querying geospatial data that changes over
25time (e.g., the growth of a city over the years due to new developments).
26
27Currently, only the geospatial features of stSPARQL have been implemented fully.
28The temporal features are the subject of current work.
29
30Given the very close relationship between stSPARQL and GeoSPARQL which is a
31recent OGC standard for an extension of SPARQL for querying geospatial metadata,
32we recently provided support for the Core, Geometry and Geometry Topology
33extension of GeoSPARQL.
34
35
36Conformance to GeoSPARQL
37========================
38Strabon implements the `Core', the `Topology Vocabulary Extension', the
39`Geometry Extension', the `Geometry Topology Extension', and the `RDFS Entailment
40Extension' except for Req. 25
41(http://www.opengis.net/spec/geosparql/1.0/req/rdfs-entailment-extension/bgp-rdfs-ent).
42
43With respect to GML, Strabon supports the GML Profile corresponding to Simple
44Features, that is, GML Simple Features Profile 2.0.
45
46
47Strabon Homepage
48================
49The homepage of Strabon is at http://www.strabon.di.uoa.gr/.
50
51
52Demo
53====
54You can find a demo of the system Strabon at http://test.strabon.di.uoa.gr/NOA/.
55
56
57How to build and run Strabon from command line
58==============================================
59Assuming you have already downloaded Strabon and you are in the top-level
60directory of Strabon, issue the following command to build it from command line:
61
62        $ mvn clean package
63
64In order to run automatically the JUnit tests, pass the option `-DskipTests=false'
65to the above command. The complete command is the following:
66
67        $ mvn -DskipTests=false clean package
68
69Alternatively, it is possible to run a specific test. Supposing that the name of
70the corresponding class is `TestName', then you can run only this
71test using the following command:
72
73        $ mvn test -DfailIfNoTests=false -DskipTests=false -Dtest=TestName
74
75In case of an error during building of Strabon and assuming that the error does
76not come from the JUnit tests, please have a look at the `Known Issues' section
77below. If none of the known issues of that section applies, please contact the
78developers through the Strabon mailing-list or submit a corresponding bug
79(see `Bugs' section below).
80
81After you have successfully built Strabon, you have access to the following
82components:
83
84  * Strabon Endpoint
85
86    This is a SPARQL endpoint for Strabon. It is distributed as a war file so
87    you may deploy it in a Tomcat container. You may find the war file under
88    directory `endpoint/target'.
89
90  * Strabon Endpoint (standalone)
91
92    This is a SPARQL endpoint for Strabon like the above one, but it differs
93    only in that it does not require the user to have already set up a Tomcat
94    container. The standalone Strabon Endpoint may be run by issuing the
95    following command:
96
97        $ java -jar endpoint-exec/target/strabon-endpoint-executable-${version}.jar
98
99    After issuing the above command, you may access the Strabon Endpoint at
100    the following URL: <http://localhost:8080/>.
101
102    Please see the page at <http://hg.strabon.di.uoa.gr/Strabon/rev/674f8f91162b>
103    to find out other options that you may pass to the Tomcat container that
104    will run by the above command.
105
106    SPECIAL NOTE: if you need to configure the connection details to the
107    underlying database, you may do so in two ways:
108      1. By modifying file `endpoint/WebContent/WEB-INF/connection.properties'
109         before building Strabon and executing the above command.
110      2. After executing the above command, by visiting the following page by a
111         browser: <http://localhost:8080/ChangeConnection>
112
113  * Strabon Endpoint Client
114
115    This is a Java client for interacting with Strabon Endpoint or any other
116    SPARQL endpoint. It is packaged as a jar file and may be found under
117    directory `endpoint-client/target/' with name
118    `strabon-endpoint-client-${version}.jar'. This jar contains any dependencies
119    to other code, so may copy and paste it to your project and start playing
120    with the code immediately.
121
122  * Strabon script
123
124    The `strabon' script is located under the `scripts/' directory and it is the
125    main command-line tool for interacting with Strabon. You may use it to store
126    RDF data with geospatial information or query/update it using one of
127    stSPARQL or GeoSPARQL query languages.
128
129  * Endpoint script
130
131    The `endpoint' script is located under the `scripts' directory and it is the
132    main command-line tool for interacting with a `Strabon Endpoint'. You may
133    use it to do any operation you would like to do with the `strabon' script
134    above, but in contrast to the `strabon' script you need to have access to a
135    Strabon endpoint. Of course, the `Strabon Endpoint Client' component above
136    can be used as well as a command-line tool. At the time of writing, the
137    `Strabon Endpoint Client' component supports only querying of RDF data with
138    geospatial information.
139
140
141Getting Started
142===============
143To get started  with Strabon please have a look at the tutorial for the stRDF
144data model and stSPARQL query language, the User Guide, and the Developer Guide.
145
146stRDF and stSPARQL tutorial
147        http://www.strabon.di.uoa.gr/files/stSPARQL_tutorial.pdf
148
149
150stSPARQL Reference
151~~~~~~~~~~~~~~~~~~
152The reference for the spatial and temporal extension functions defined in
153stSPARQL can be found at http://www.strabon.di.uoa.gr/stSPARQL#spatial and
154http://www.strabon.di.uoa.gr/stSPARQL#temporals respectively.
155
156
157User Guide
158~~~~~~~~~~
159Assuming that you are familiar with Maven, the following steps need to be
160followed in order to use Strabon using Eclipse:
161
1621. Install PostgreSQL from http://www.postgresql.org/download/. At the time of
163   this writing the latest PostgreSQL version is 9.1.
1642. Install PostGIS from http://postgis.refractions.net/download/. At the time of
165   this writing we have tested Strabon with PostGIS 1.5.3.
1663. Install Maven from http://maven.apache.org/download.html. At the time of this
167   writing the latest Maven version is 3.0.4.
1684. Install Eclipse from http://www.eclipse.org/downloads/. At the time of this
169   writing the latest Eclipse version is 3.7.2.
1705. Install the m2e plugin for Eclipse from http://www.eclipse.org/m2e/.
1716. Install the MercurialEclipse plugin for Eclipse from
172   http://javaforge.com/project/HGE .
1737. From Eclipse, go to File --> Import --> Mercurial --> Clone Existing
174   Mercurial Repository --> Next. In the URL textarea paste the following
175   URL: http://hg.strabon.di.uoa.gr/StrabonUser and then press Next --> Next -->
176   Finish. If you used the default settings, you should have a new project named
177   StrabonMain. Right click on the project and select Configure --> Convert to
178   Maven project. Eclipse will enable Maven dependency management for the
179   project, download any dependencies and build the project.
180
181
182        Storing stRDF graphs and evaluating stSPARQL queries
183        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
184You can see some examples in the classes gr.uoa.di.strabon.example.PostgisExample
185and gr.uoa.di.strabon.example.PostgisExample2.
186
187        Tuning PostgreSQL
188        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
189The default settings of Postgres are rather conservative. As a result, parameter
190tuning is neccessary for speeding up Postgres, therefore Strabon. If you are
191using Strabon to compare its performance against your implementation of
192stSPARQL/GeoSPARQL, you are *strongly* encouraged to contact us using the Strabon
193Users mailing list for assistance on tuning Postgres.
194
195You can follow the instructions below for tuning a Postgres server running on an
196Ubuntu machine that is dedicated to PostgreSQL and Strabon.
197
1981. Append the following text at the end of postgresql.conf.
199*Uncomment* the appropriate lines.
200
201### RAM
202## 4 GB of RAM
203#shared_buffers       =  3GB
204#effective_cache_size =  3GB
205#maintenance_work_mem =  1GB
206#work_mem             =  2GB
207## 8 GB of RAM
208#shared_buffers       =  5GB
209#effective_cache_size =  6GB
210#maintenance_work_mem =  2GB
211#work_mem             =  5GB
212## 16 GB of RAM
213#shared_buffers       = 10GB
214#effective_cache_size = 14GB
215#maintenance_work_mem =  4GB
216#work_mem             = 10GB
217## 24 GB of RAM
218#shared_buffers       = 16GB
219#effective_cache_size = 22GB
220#maintenance_work_mem =  6GB
221#work_mem             = 15GB
222## 48 GB of RAM
223#shared_buffers       = 32GB
224#effective_cache_size = 46GB
225#maintenance_work_mem =  8GB
226#work_mem             = 30GB
227## 64 GB of RAM
228# contact us to find out!
229### HD
230## RAID with ordinary 7.200 disks
231#random_page_cost = 3.5 #3.0-3.5
232## High-End NAS/SAN
233#random_page_cost = 2 #1.5-2.5
234## Amazon EBS/Heroku
235#random_page_cost = 1.3 #1.1-2.0
236## SSD array
237#random_page_cost = 2.0 #1.5-2.5
238### Planner options
239# Increase the following values in order to avoid using the GEQO planner.
240# Small values (<8) reduce planning time but may produce inferior query plans
241#
242geqo_threshold = 15 # keep this value larger that the following two parameters
243from_collapse_limit = 14
244join_collapse_limit = 14
245### Misc
246default_statistics_target    = 10000
247constraint_exclusion         = on
248checkpoint_completion_target = 0.9
249wal_buffers                  = 32MB
250checkpoint_segments          = 64
251### Connections
252max_connections              = 10
253
2542. Append the following lines at the end of /etc/sysctl.conf
255*Uncomment* the appropriate lines.
256
257## 4 GB of RAM
258#kernel.shmmax = 3758096384
259#kernel.shmall = 3758096384
260#kernel.shmmni = 4096
261## 8 GB of RAM
262#kernel.shmmax = 5905580032
263#kernel.shmall = 5905580032
264#kernel.shmmni = 4096
265## 16 GB of RAM
266#kernel.shmmax = 11274289152
267#kernel.shmall = 11274289152
268#kernel.shmmni = 4096
269## 24 GB of RAM
270#kernel.shmmax = 17716740096
271#kernel.shmall = 17716740096
272#kernel.shmmni = 4096
273## 48 GB of RAM
274#kernel.shmmax = 35433480192
275#kernel.shmall = 35433480192
276#kernel.shmmni = 4224
277## 64 GB of RAM
278# contact us to find out!
279
2803. Apply all changes by executing
281
282$ sudo sysctl -p
283$ sudo /etc/init.d/postgresql restart
284
2854. Prepare for the next run by issuing the command
286
287$ sudo -u postgres psql -c 'VACUUM ANALYZE;' db
288
289or
290
291$ psql -c 'VACUUM ANALYZE;' db
292
293where db is the name of the Postgres database that Strabon will use.
294
295
296Developer Guide
297~~~~~~~~~~~~~~~
298Assuming that you are familiar with Maven, the following steps need to be
299followed in order to use Strabon using Eclipse:
300
3011. Install PostgreSQL from http://www.postgresql.org/download/. At the time of
302   this writing the latest PostgreSQL version is 9.1.
3032. Install PostGIS from http://postgis.refractions.net/download/. At the time of
304   this writing we have tested Strabon with PostGIS 1.5.3.
3053. Install Maven from http://maven.apache.org/download.html. At the time of this
306   writing the latest Maven version is 3.0.4.
3074. Install Eclipse from http://www.eclipse.org/downloads/. At the time of this
308   writing the latest Eclipse version is 3.7.2.
3095. Install the m2e plugin for Eclipse from http://www.eclipse.org/m2e/.
3106. Install the MercurialEclipse plugin for Eclipse from
311   http://javaforge.com/project/HGE.
3127. From Eclipse, go to File --> Import --> Mercurial --> Clone Existing
313   Mercurial Repository --> Next. In the URL textarea paste the following
314   URL: http://hg.strabon.di.uoa.gr/Strabon and then press Next --> Next -->
315   Finish. If you used the default settings, you should have a new project named
316   StrabonMain. Right click on the project and select Configure --> Convert to
317   Maven project. Eclipse will enable Maven dependency management for the
318   project, download any dependencies and build the project.
319
320
321Tester Guide
322~~~~~~~~~~~~
323Assuming again that you are familiar with Maven and Junit these are the steps
324you need to follow to test the functionality of Strabon:
325
326
327* Using Eclipse
328
329        If you want to create a new test:
330
331        1. Import Strabon into Eclipse as explained in the Developer Guide.
332        2. Go to strabon-testsuite project.
333        3. Create a new folder (Recommended folder name: <test's name>) and place inside the following files:
334                3.1. An ntriples or nquads file with the test dataset (with .nt or .nq extension).
335                3.2. Pairs of files with sparql test queries and expected test results in xml format.
336                         Notice that each pair must have the same name and .rq extension for the queryFile and .srx extension for the resultsFile.
337        4. Create a test class that extends TemplateTest class.
338        5. If you have followed the recommendations the test is ready. If you have different names or location for your
339        files, insert them explicitly in the constructor of the class. WARNING: All prefixes must be placed in file "prefixes" so that
340        every time a namespace changes, we have to change it just once.
341       
342        If you want to run a test:
343       
344        1. Right-click on the test class.
345        2. Select "Run as JUnit Test".
346        3. Database properties are retrieved from database.properties file. If you want, you can change a property
347        "on the fly" with an environment variable.
348
349       
350* Command Line
351       
352        If you want to run all the tests:
353       
354        1. Go to Strabon directory (root directory of all the subprojects).
355        2. Run "mvn test -DskipTests=false".
356        3. Optionally you can pass an environment variable with "-DvariableName=variableValue".
357
358
359Storing stRDF graphs and evaluating stSPARQL queries
360~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
361You can see some examples in the classes
362eu.earthobservatory.runtime.postgis.StoreOp and
363eu.earthobservatory.runtime.postgis.QueryOp.
364
365
366Download
367========
368You can download the source code of the latest version of Strabon by accessing
369our public mercurial repository located at http://hg.strabon.di.uoa.gr/Strabon.
370You can find more information on how to use and extend Strabon at the Getting
371Started section.
372
373
374Publications
375============
376You can learn about stRDF data model and stSPARQL query language employed in
377Strabon by reading our tutorial under the Getting Started section and/or the
378publications given on this page.
379
380The current versions of stRDF and stSPARQL which are based on OGC standards are
381presented in the following document:
382  * K. Kyzirakos, M. Karpathiotakis, and M. Koubarakis. Strabon: A Semantic
383    Geospatial DBMS. In Internatioanl Semantic Web Conference (ISWC'12). Boston,
384    USA, November 11-15, 2012.
385    [pdf: http://strabon.di.uoa.gr/files/strabon-iswc.pdf]
386
387  * Manolis Koubarakis, Kostis Kyzirakos, Babis Nikolaou, Michael Sioutis, and
388    Stavros Vassos. A data model and query language for an extension of RDF with
389    time and space. Deliverable D2.1, European ICT project TELEIOS, 2011.
390    [pdf: http://strabon.di.uoa.gr/files/deliv2-1-re-revised.pdf]
391
392The initial versions of stRDF and stSPARQL that are based on constraint
393databases are presented in the following publications:
394  * Manolis Koubarakis and Kostis Kyzirakos. Modeling and Querying Metadata in
395    the Semantic Sensor Web: the Model stRDF and the Query Language stSPARQL.
396    In 7th Extended Semantic Web Conference (ESWC 2010). Heraklion, Crete,
397    30 May - 03 June, 2010.
398    [pdf: http://strabon.di.uoa.gr/files/stSPARQL.pdf]
399 
400  * Kostis Kyzirakos, Manos Karpathiotakis and Manolis Koubarakis. Developing
401    Registries for the Semantic Sensor Web using stRDF and stSPARQL (short
402    paper).
403    In Proceedings of 3rd International workshop on Semantic Sensor Networks
404    2010, in conjunction with ISWC 2010, November 2010, Shanghai, China.
405    [pdf: http://strabon.di.uoa.gr/files/strabon.pdf]
406
407Applications of stRDF, stSPARQL, and the system Strabon are described here:
408  * Alasdair J. G. Gray, Raúl García-Castro, Kostis Kyzirakos, Manos
409    Karpathiotakis, Jean-Paul Calbimonte, Kevin Page, Jason Sadler, Alex
410    Frazer, Ixent Galpin, Alvaro A. A. Fernandes, Norman W. Paton, Oscar
411    Corcho, Manolis Koubarakis, David De Roure, Kirk Martinez and Asunción
412    Gómez-Pérez. A Semantically Enabled Service Architecture for Mashups over
413    Streaming and Stored Data. In 8th Extended Semantic Web Conference (ESWC
414    2011). Heraklion, Crete, May 20 - June 2, 2011.
415    [pdf: http://strabon.di.uoa.gr/files/Gray2011Architecture.pdf]
416
417  * A.J.G. Gray, J. Sadler, O. Kit, K. Kyzirakos, M. Karpathiotakis, J.-P.
418    Calbimonte, K. Page, R. García-Castro, A. Frazer, I. Galpin, A.A.A.
419    Fernandes, N.W. Paton, O. Corcho, M. Koubarakis, D.D. Roure, K. Martinez,
420    A. Gómez-Pérez. A Semantic Sensor Web for Environmental Decision Support
421    Applications. Sensors. 11, 8855-8887.
422    [pdf: http://strabon.di.uoa.gr/files/sensors-11-08855.pdf]
423
424Coming up soon:
425Strabon will soon support an extension of RDF for incomplete geospatial
426information. The following publication gives a preview of the relevant research
427problems:
428  * M. Koubarakis, K. Kyzirakos, M. Karpathiotakis, C. Nikolaou, M. Sioutis,
429    S. Vassos, D. Michail, T. Herekakis, C. Kontoes and I. Papoutsis. Challenges
430    for Qualitative Spatial Reasoning in Linked Geospatial Data. In Proceedings
431    of IJCAI 2011 Workshop on Benchmarks and Applications of Spatial Reasoning,
432    Barcelona, Spain.
433    [pdf: http://www.earthobservatory.eu/publications/SciQL_ADASS2011.pdf]
434
435  * C. Nikolaou and M. Koubarakis: "Querying Linked Geospatial Data with
436    Incomplete Information". In 5th International Terra Cognita Workshop -
437    Foundations, Technologies and Applications of the Geospatial Web. In
438    conjunction with the 11th International Semantic Web Conference, Boston,
439    USA, November, 2012.
440    [http://www.earthobservatory.eu/publications/iswc-workshop.pdf]
441
442
443Contributors
444============
445The system Strabon has been developed by the following members of our team:
446
447* Manos Karpathiotakis  <mk@di.uoa.gr>
448* Kostis Kyzirakos      <Kostis.Kyzirakos@cwi.nl>
449* Manolis Koubarakis    <koubarak@di.uoa.gr>
450* Giorgos Garbis        <ggarbis@di.uoa.gr>
451* Konstantina Bereta    <konstantina.bereta@di.uoa.gr>
452* Charalampos Nikolaou  <charnik@di.uoa.gr>
453* Stella Gianakopoulou  <sgian@di.uoa.gr>
454* Panayiotis Smeros     <psmeros@di.uoa.gr>
455* Kallirroi Dogani      <kallirroi@di.uoa.gr>
456
457
458Mailing-list
459============
460Currently, we maintain the following mailing lists:
461
462* Strabon-users, is used as a communication channel for Strabon users.
463To subscribe to the mailing-list, please visit page
464http://cgi.di.uoa.gr/~mailman/listinfo/strabon-users. To post e-mails
465to Strabon-users mailing-list, write to strabon-users@di.uoa.gr.
466
467* Strabon-devel, is used as a communication channel with the developers
468of Strabon. To subscribe to the mailing-list, please visit page
469http://cgi.di.uoa.gr/~mailman/listinfo/strabon-devel. To post e-mails
470to Strabon-devel mailing-list, write to strabon-devel@di.uoa.gr.
471
472
473Bugs
474====
475Please report bugs to http://bug.strabon.di.uoa.gr/report or
476the Strabon-devel mailing-list Strabon-devel@di.uoa.gr.
477
478
479Known Issues
480============
481 * By default, Tomcat uses ISO-8859-1 character encoding when decoding URLs received
482   from a browser. This can cause problems when encoding is UTF-8, and you are using
483   international characters. In order to fix this, edit conf/server.xml and find the
484   line where the Connector is defined. Add the parameter URIEncoding and set it to
485   UTF-8. For example:
486
487   <Connector port="8080" protocol="HTTP/1.1"
488                  connectionTimeout="20000"
489                  URIEncoding="UTF-8"
490                  redirectPort="8443" />
491
492 * Building and executing any maven goals fails for maven versions <3.0 due to a
493   dependency to the `shade' plugin that is available only for maven version 3.0
494   (http://maven.apache.org/plugins/maven-shade-plugin/). In such systems, you may
495   disable execution of this plugin by setting the environmental variable
496   `shade.skip'. For example, to build Strabon using maven version 2.0 you may
497   execute the following command:
498
499        $ mvn clean package -Dshade.skip
500
501 * When using MonetDB as a backend, the following source code of MonetDB must be
502   used:
503        https://hg.strabon.di.uoa.gr/MonetDB/   
504
505
506License
507=======
508This Source Code Form is subject to the terms of the Mozilla Public
509License, v. 2.0. If a copy of the MPL was not distributed with this
510file, You can obtain one at http://mozilla.org/MPL/2.0/.
511
512Copyright (C) 2010, 2011, 2012, Pyravlos Team
513
514http://www.strabon.di.uoa.gr/
515
516
517How to apply the license
518========================
519 * In the beginning of script files (after the shell directive) paste the
520   following statement:
521#
522# This Source Code Form is subject to the terms of the Mozilla Public
523# License, v. 2.0. If a copy of the MPL was not distributed with this
524# file, You can obtain one at http://mozilla.org/MPL/2.0/.
525#
526# Copyright (C) 2010, 2011, 2012, Pyravlos Team
527#
528# http://www.strabon.di.uoa.gr/
529#
530
531 * In the beginning of Java source code files paste the following statement:
532/**
533 * This Source Code Form is subject to the terms of the Mozilla Public
534 * License, v. 2.0. If a copy of the MPL was not distributed with this
535 * file, You can obtain one at http://mozilla.org/MPL/2.0/.
536 *
537 * Copyright (C) 2010, 2011, 2012, Pyravlos Team
538 *
539 * http://www.strabon.di.uoa.gr/
540 */
541
542 * In the beginning of HTML/XML files paste the following statement:
543<!-- This Source Code Form is subject to the terms of the Mozilla Public
544   - License, v. 2.0. If a copy of the MPL was not distributed with this
545   - file, You can obtain one at http://mozilla.org/MPL/2.0/.
546   -
547   - Copyright (C) 2010, 2011, 2012, Pyravlos Team
548   -
549   - http://www.strabon.di.uoa.gr/
550-->
Note: See TracBrowser for help on using the repository browser.