ies-svn.jrc.ec. Web viewfor comments . Reviewer: ARE3NA ISA action. Date: 2016-03-11. Document:...
38
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03- 11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T ATS (e.g. download -atom) Test (e.g. A.01.TGR1) Type of comment 1 Severity (minor, medium, critical) Comments Proposed change Resolution all * GE Critical Based on ISO 19105 and the OGC Specification Model, each Markdown document represents a test case. A test case is part of a conformance class, which in turn is part of an abstract test suite for a specification document. The current material seems to use “ATS” where “Conformance Class” is probably meant. The information which conformance classes form an ATS is, however, not represented. In this context it is important to emphasize that conformance class is the key concept. An ATS is far less important as it is just the aggregation of all the conformance classes in a specification. In addition, the term “ATS” is sometimes also used for a test case. Use “Conformance Class” (or “CC”/“cc”) where “Abstract Test Suite” (or “ATS”/“ats”) is used now - and where the Technical Guidance includes (implicitly or explicitly) a conformance class. Avoid using wrong a misleading term for test cases. Group the conformance classes into ATSs based on the specifications that define the conformance classes. I.e., the structure would be for the current “ATS”s: ATS: TG View Service 3.11 o CC: WMS 1.3.0 profile TC: A.02.IR04.extended... TC: A.03.IR05.schema... TC: ... (no more test cases shown below) o CC: WMTS 1.0.0 profile ATS: TG Discovery Service 3.1 o CC: CSW 2.0.2 ISO AP profile ATS: TG Download Service 3.1 o CC: Pre-defined Atom o CC: Pre-defined WFS 2.0.0 o CC: Direct WFS 2.0.0 1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test) CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement) page 1 of 38
Transcript of ies-svn.jrc.ec. Web viewfor comments . Reviewer: ARE3NA ISA action. Date: 2016-03-11. Document:...
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
all * GE Critical Based on ISO 19105 and the OGC Specification Model, each Markdown document represents a test case. A test case is part of a conformance class, which in turn is part of an abstract test suite for a specification document.The current material seems to use “ATS” where “Conformance Class” is probably meant. The information which conformance classes form an ATS is, however, not represented.In this context it is important to emphasize that conformance class is the key concept. An ATS is far less important as it is just the aggregation of all the conformance classes in a specification.In addition, the term “ATS” is sometimes also used for a test case.
Use “Conformance Class” (or “CC”/“cc”) where “Abstract Test Suite” (or “ATS”/“ats”) is used now - and where the Technical Guidance includes (implicitly or explicitly) a conformance class. Avoid using wrong a misleading term for test cases.Group the conformance classes into ATSs based on the specifications that define the conformance classes.I.e., the structure would be for the current “ATS”s:
ATS: TG View Service 3.11
o CC: WMS 1.3.0 profile
TC: A.02.IR04.extended...
TC: A.03.IR05.schema...
TC: ... (no more test cases shown below)
o CC: WMTS 1.0.0 profile
ATS: TG Discovery Service 3.1
o CC: CSW 2.0.2 ISO AP profile
ATS: TG Download Service 3.1
o CC: Pre-defined Atom
o CC: Pre-defined WFS 2.0.0
o CC: Direct WFS 2.0.0
o CC: Quality of Service
ATS: TG Metadata 1.3
o CC: ISO 19115/19119 profile
ATS: TG Spatial Data Services 3.1
o CC: Invocable Spatial Data Services
o CC: Interoperable Spatial Data Services
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 1 of 24
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
o CC: Harmonized Spatial Data Services
Notes:
TG View Services: The document does not explicitly specify conformance classes for the two OGC standards, but implicitly these are defined as “profiles”.
TG View and Discovery Services: It is unclear how the requirements regarding QoS have to be treated as the requirements are not identified as requirements with an identifier.
all * GE Critical An ATS belongs to a specification. Therefore, the references in a test case may only reference one specification.However, currently sometimes an IR and a TG are referenced. This is not consistent with the notion of conformance classes or abstract test suites. Since IRs do not specify conformance classes and this is typically done in a technical guidance document, the reference should go to the TG (and the requirement in the TG should reference the relevant parts of the IR).
For each test case, identify/reference the relevant requirements in the TG document that specifies the conformance class.
For TGs that identify requirements, list the requirements, otherwise list the most specific section.
all * GE Critical For cases where a dependency exists to an external conformance class (dependencies are always on the conformance class level, not on the level of test suites or test cases), clearly identify the conformance class.
See comment. For example, do not reference tests “OGC FES 2.0, A.1 Test cases for query”, but reference OGC FES 2.0 Conformance Class “Query”.
all * GE Critical It would be very useful to clarify for each test case which messages should be reported in case of identified errors in the test object.While the description of a test case should be independent from both the implementation and the values, the test case “should be complete in the sense that it is sufficient to enable a test verdict to be assigned unambiguously to each potentially observable test outcome (i.e. sequence of test events)” (see ISO 19105). This level of completeness would imply that the potential verdicts / messages can be derived easily from
For each test case, specify the messages (including the variable information derived from the test object) that should be reported.
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 2 of 24
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
the test case description.In some cases this is trivial, but in general there is a risk that the messages implemented in an ETS may not contain the information that the ATS authors expect.
all * GE Critical The section “Prerequisites” should have a reliable structure. Prerequisites should only be other test cases in the same conformance class or other conformance classes.In some cases, this section does not state other tests that need to be passed first, but actually state new tests.
Update “Prerequisites” to be a list of other test cases in the same conformance class or other conformance classes.
Any prerequisites that actually are new tests need to be converted into separate test cases or included in the test method description.
all * GE Critical The documentation of the test cases is inconsistent. Sometimes the title is the label (A.xxx.xxx) and sometimes a text.
Make the test case documentation consistent.
all * GE Critical The test type is mostly “automated”, but some test are “manual” or it is unclear. However, in some cases a requirement is considered “not testable” where a manual test might be possible.It is clear that the ETS will only implement automated tests, but it may be helpful to understand if manual tests are within the scope of the ATSs or not.
Clarify.
download-atom
A.02.TGR2.conformtoAtomSpecification
CT / GE Critical Is this only about validation against RelaxNG or checking conformance against RFC 4287?
If the latter, conformance to RFC 4287 should be a separate ATS and identify the test cases to a level that unambiguously identifies the assertions to test.
download-atom
A.03.TGR3.conformtoGeoRSS-Simple
CT / GE Critical Is this only about validation of selected elements in an Atom feed against the GeoRSS XML Schema or checking conformance against the GeoRSS specification?
If the latter, conformance to GeoRSS should be a separate ATS and identify the test cases to a level that unambiguously identifies the assertions to test.
download-atom
A.04.TGR4.conformtoOpenSearch1.1
CT / GE Critical Validation of OpenSearch description document is more than a test case. There is no schema document or similar that could be used.(If a single test case would be ok here, there would be no need to break INSPIRE conformance classes down into test cases, just one test case “test conformance against INSPIRE Technical Guidance” would be sufficient.)
Define ATS for the OpenSearch description and identify the test cases to a level that unambiguously identifies the assertions to test.
download-atom
A.04.TGR4.conformtoOpenSearch1.1
GE Medium Open Search 1.1 is not a finalised specification. http://www.opensearch.org/Specifications/OpenSearch/1.1 redirects to Draft 5.
Clarify which draft version is meant (probably Draft 5).
download- A.06.IR511.TG CT Medium “the INSPIRE language request parameters in the Clarify.
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 3 of 24
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
atom R6.linkToMetadataForTheService
Resource Locator to the Atom Download Service Feed URLs must be ignored in the comparison” is unclear. Which comparison? The does not seem to be any comparisons of feed URLs only of metadata record URLs?
download-atom
A.06.IR511.TGR6.linkToMetadataForTheService
CT Minor “valid gmd:MD_Metadata element” is most likely understood to mean “schema valid”.
If something else is meant, the test should be clarified.
download-atom
A.07.TGR7.selfreference
CT Medium Test method seems incomplete. Clarify Xpath reference for “he default language code defined in the OpenSearch description”.
download-atom
A.08.IR222.TGR8.linktoOpenSearchDescription
GE Medium How is this different from A.04.TGR4.conformtoOpenSearch1.1?
Drop one test.
download-atom
A.09.TGR9.feedid, A.21.TGR22.datasetFeedId
CT Minor Strictly, the test differs from the requirement. The requirement is not that the id is the same the feed URI provided, but that the id resolves to the same document.
download-atom
A.10.IR221.TGR10.rightselement
ED Minor Regarding note 1, the requirement is clear that only /feed/rights is covered.
Remove note 1.
download-atom
A.11.IR221.TGR11.updatedelement, A.23.IR221.TGR24.datasetFeedUpdated
CT Medium “... or too far in the past” is vague for testing. Change to “... or before 2012 (first release of the Technical Guidance)“?
download-atom
A.12.IR221.TGR12.contactinformation, A.24.IR221.TGR25.datasetFeedContactinformation
CT Medium Unresolved issue: “A regular expression could be used to validate the email address. Several regular expressions are available; the workgroup could choose one.”
Delete or resolve.
download-atom
A.13.IR221.TGR13.datasetidentifiers
CT Medium „the dataset identifier code and dataset identifier namespace must be present in the metadata document of the service“ is too vague. What does „be present“ mean? Included somewhere in the document or in specific elements?
Clarify.
download-atom
A.14.IR221.TGR14.linksToDatasetMetadata
CT Medium Same issue as with A.13.IR221.TGR13.datasetidentifiers (“present in the metadata document of the service and in the
Clarify.
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 4 of 24
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
Download Service feed”). What does present mean?
download-atom
A.14.IR221.TGR14.linksToDatasetMetadata
CT Medium gmd:identificationInfo[1]/*/gmd:citation/*/gmd:identifier may refer to multiple nodes.
Clarify how to deal with multiple identifiers in the test.
download-atom
A.14.IR221.TGR14.linksToDatasetMetadata
CT Medium gmd:identificationInfo[1]/*/gmd:citation/*/gmd:identifier may be gmd:MD_Identifier, RS_Identifier or some other element.
Clarify how to compare identifiers that use different data types.
download-atom
A.18.TGR19.entryUpdated
CT Medium Why is the updated test different from A.11.IR221.TGR11.updatedelement? (Any year will work here.)
Consider aligning tests.
download-atom
A.26.IR313.TGR27.separateEntriesCRSFormat
CT Medium “Find and retrieve the all the Download Service Feed documents containing an entry pointing to this Dataset Feed”.Why “all” the Download Service feed documents? As the test object is “a” Download Service this test should only check the feed of the Download Service under test.Consistent with this, the Download Service feed document accessed earlier must be used not any other feed document.
“For each category element in the Download Service feed entity, which included the link to the Dataset feed document, check that at least one entry exists in the Dataset feed containing a category element with an identical term attribute.”
download-atom
A.25.IR31.TGR26.datasetFeedDownloadLink, A.28.IR31.TGR29.datasetFeedDownloadLinkDetails
CT Medium These seem to overlap significantly. Consider to merge both test cases.
download-atom
A.29.IR311.TGR31.languageForDownloadLink
CT Medium Unresolved issue: “Is the hreflang attribute still mandatory if data in only 1 language is provided? If not, this ATS is not automatically testable and the ATS should be removed.”
Clarify.
download-atom
A.34.IR222.TGR39.provideOpenSearchDescription
GE Minor This test case is not referenced from the overview and not testable.
Remove test case.
download-atom
A.36.TGR41.openSearchGenericSearchQueries,A.37.IR4.TGR42.openSearc
CT Medium „test if it provides a document with content-type xxx’“ is vague. Is this a test on the HTTP header of the response or in some way a test of the response.
Clarify. Probably a test on the HTTP header of the response is meant.
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 5 of 24
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
hUrlDescribeSpatialDataset
download-atom
A.39.IR3.IR4.TGR44.openSearchQueryExample
CT Minor valid HTTP codes: “200,206,301,303,303” Change first 303 to 302.
download-predefined-wfs
* ED Minor Numbering of test cases jumps from A.04 to A.06 Update numbering of test cases
download-predefined-wfs
A.02.IR2.IR4.TGR49.TGR50.TGR51.predefinedStoredQuery
CT Critical “For each combination of supported CRS, supported language, SpatialDataSetIdentifier ID code and SpatialDataSetIdentifier Namespace”The mechanism to determine the “supported CRS” is problematic as the CRS information may differ from feature type to feature type. It is unclear in how far that is an issue in practice, but a conformant service may specify different CRSs per feature type.
See comment. As the TG is unclear, the requirement should be clarified in the TG first.
download-predefined-wfs
A.03.IR221.TGR53.serviceMetadata
CT Medium The test method is not consistent with the purpose. The test method checks that both all metadata elements are in the extended capabilities and that there is a MetadataURL pointing to a valid Metadata document.
Change test method to express the ”EITHER ... OR”.
download-predefined-wfs
A.03.IR221.TGR53.serviceMetadata
CT Medium “valid Metadata document” is most likely understood to mean “schema valid”. If something else is meant, which is likely as schema validity does not make the metadata document conformant with the service metadata requirements, the test should be clarified.
Clarify meaning of “valid”.
download-predefined-wfs
A.03.IR221.TGR53.serviceMetadata
CT Minor Note also that the TG seems to be incorrect and the reference to table 4 should be to table 19.
Update the TG to reference table 19, not table 4.
download-predefined-wfs
A.04.TGR55.TGR56.language.affects.capabilities
CT Medium VERSION is not a parameter of the GetCapabilities request.
Change to ACCEPTVERSIONS.
download-predefined-wfs
A.04.TGR55.TGR56.language.affects.capabilities
CT Critical “If the returned resource can be parsed as a valid XML document and if the document passes all the tests listed as prerequisites for this test”Why do the tests need to be executed again, if they have been tested before (“prerequisites”)?The test cases and conformance classes in “prerequisites” have a WFS 2.0.0 as the test
Either drop the prerequisites and explicitly state the steps/assertions or omit the sentence.
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 6 of 24
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
object, while this statement is worded as if these would be test cases and conformance classes on a Capabilities document.
download-predefined-wfs
AT Minor Maybe a test for requirement 52 / 61 could be added by testing that for each feature type there is exactly one //wfs:FeatureType/wfs:MetadataURL and all values must be identical.However, the TG does not seem to require this, so this would require a TG update first.
-
ats-download-directaccess-wfs
* AT Minor Maybe a test for requirement 52 / 61 could be added by testing that for each feature type there is exactly one //wfs:FeatureType/wfs:MetadataURL and all values must be identical.However, the TG does not seem to require this, so this would require a TG update first.
-
ats-metadata
A.01.validate CT minor The ATS requires validation against three different XSD schema versions (at least one should pass). It is not clear how to determine which schema to use to validate the document.
The ATS should perhaps clarify that this is on the basis of the declared namespaces in the XML document (see also issue 10 on GitHub), i.e.:If the document declares http://www.isotc211.org/2005/srv, use http://schemas.opengis.net/csw/2.0.2/profiles/apiso/1.0.0/apiso.xsd.But otherwise, it is unclear what to use. The official schema for http://www.isotc211.org/2005/gmd is http://standards.iso.org/ittf/PubliclyAvailableStandards/ISO_19139_Schemas/gmd/gmd.xsd or http://schemas.opengis.net/iso/19139/20070417/gmd/gmd.xsd, but in the context of Discovery services, INSPIRE also uses http://schemas.opengis.net/iso/19139/20060504/gmd/gmd.xsd. It is unclear how to select the schema document. Which schema document does the MD TG require?
ats-metadata
A.02.title CR minor There is still an open question whether to include this test case, as there is no explicit requirement in the technical guidelines.
It is proposed to keep this test case, and change the technical guidelines (TG MD) including this requirement.
ats-metadata
A.03.abstract CR minor There is still an open question whether to include this test case, as there is no explicit requirement in the technical guidelines.
It is proposed to keep this test case, and change the technical guidelines (TG MD) including this requirement.
ats-metadata
A.05.IR14.ds.keyword
ED Another prerequisite is test case A.04. Add the test case as a prerequisite.
ats- A.05.IR14.ds.k CT minor Coverage: In theory, the single keyword referring
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 7 of 24
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
metadata eyword to the INSPIRE data theme could also be a text value, in each of the official languages. Is it OK to assume that in practice all metadata providers will use the language-neutral code?
ats-metadata
A.06.IR15.srv.keyword
ED Another prerequisite is test case A.04, needed to determine whether the resource is a service.
Add the test case as a prerequisite.
ats-metadata
A.07.IR05.IR06.ds.identification
CR,CT
Ambiguity: The discussion seems to be still ongoing: “In case of MD_identifier, discussion is ongoing on how to match this element against a namespace-identifier in a capabilities document/service metadata.” (see issue MIWP-8 (L) Unique Resource Identifier)
ats-metadata
A.07.IR05.IR06.ds.identification
ED Ambiguity: gmd:identificationInfo[1]/*/gmd:citation/*/gmd:identifier may refer to multiple nodes. It should be clarified how to deal with multiple identifiers in the test.
Note: Another prerequisite is test case A.04.
Clarify ambiguity and add test case as a prerequisite.
ats-metadata
A.08.IR03.ds.linkage
CT Ambiguity: It is not entirely clear which parts of the WSDL or GetCapabilities document need to be tested: “If the response indicates a linkage is a service capabilities or WSDL document, some basic params in the service response are analysed”. Or does this only refer to “Any service response should be checked if it provides proper linkage. The service wsdl or capabilities document should have a featuretype that shares the resource unique identification.”
Perhaps CI_OnLineFunctionCode (optional element) could be used to determine the nature of the online resource locator if present.
ats-metadata
A.08.IR03.ds.linkage
CT Testability: A manual test is suggested, if the resource locator is a web page with further instructions or a client application.
ats-metadata
A.09.IR04.srv.linkage
CT Ambiguity: “The URL is resolved.” It is not entirely clear what resolving the URI means. Which maximum timeout can be applied? Should we only look at the HTTP status code (200), or also at the returned information?Ambiguity: It is not entirely clear which parts of the WSDL or GetCapabilities document need to be tested: “If the response indicates a linkage is a service capabilities or WSDL document, some basic params in the service response are analysed.” Or does it only refer to: “Any service response should be checked if it provides proper
Specify which HTTP status codes are acceptable.
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 8 of 24
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
linkage. The service wsdl or capabilities document should have a featuretype that shares the resource unique identification.”Testability: A manual test is suggested, if the resource locator is a web page with further instructions or a client application.Note: Perhaps CI_OnLineFunctionCode (optional element) could be used to determine the nature of the online resource locator if present (see also TG SDS req 3).Note: What in case of access restrictions?Note: Another prerequisite is test case A.04.
ats-metadata
A.10.IR08.IR09.ds.language
Coverage: The test should not go as far as testing whether there are textual values in the datasets or data series.Ambiguity: The test should specify what the “valid 3-letter language codes according to ISO/TS 19139” are.Note: The test could perhaps explicitly test if the code list is identified by the URI: http://www.loc.gov/standards/iso639-2/ . The US Library of Congress is the officially mandated organisation to maintain ISO639.Note: Another prerequisite is test case A.04.
ats-metadata
A.11.IR10.IR11.ds.topic
ED Another prerequisite is test case A.04. Add the test case as a prerequisite.
ats-metadata
A.12.IR12.srv.type
ED Another prerequisite is test case A.04. Add the test case as a prerequisite.
ats-metadata
A.14.IR16.IR17.IR18.vocab
CT Testability: Validating if the keyword is actually available in the indicated vocabulary is a challenge, since the vocabulary is usually not referenced by a URL. If a vocabulary is indicated that is available to the validator, then this check can be performed.
ats-metadata
A.16.IR20.IR21.ds.bounds
CT Testability: The bounding box shall be as small as possible. Quite hard to honour. Data should be downloaded and a minimal bounds could be calculated and compared to the indicated bounds.Furthermore, data could be access controlled or in an unknown format. This would hinder testability further. Lastly, this may also require coordinate transformations as the source data may be in a projected CRS or use a different Meridian.
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 9 of 24
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
ats-metadata
A.16.IR20.IR21.ds.bounds
ED Another prerequisite is test case A.04. Add the test case as a prerequisite.
ats-metadata
A.17.IR22.IR23.ds.temporal
CT medium As formulated now, the test result does not depend on the outcome of the first test on TimePeriod, so even in case of an invalid TimePeriod, the test will pass.Furthermore, the test refers to the “validity” of the date (ISO 8601 date format), which strictly speaking relates to MD TG requirement 24.
Perhaps it makes sense to split up into two test cases, one for IR22 and one for IR23.Add a specific test case for MD TG requirement 24.
ats-metadata
A.22.IR33..IR34.ds.access.use
CT Testability: This cannot be tested in an automated way: “Descriptions of terms and conditions, including where applicable, the corresponding fees shall be provided through this element or a link (URL) where these terms and conditions are described.”
ats-metadata
A.22.IR33..IR34.ds.access.use
CR The texts ‘no conditions apply’ and ‘conditions unknown’ may be replaced by language neutral codes. See: MIWP-8 (I) Language neutral identifiers.
ats-metadata
A.24.responsible.party.role
CR There is still an open question on whether to include this test case, as there is no explicit requirement in the technical guidelines.
It is proposed to keep this test case as the element tested is a mandatory one. Therefore the technical guidelines (TG MD) including this requirement should be changed.
ats-metadata
A.28.creation.date
CR This test case was removed from GitHub?There is still an open question whether to include this test case, as there is no explicit requirement in the technical guidelines.
It is proposed to add an explicit requirement for metadata date in the MD TG. A corresponding test case must also be created.
ats-metadata
A.29.IR07.srv.identification
CR medium Open issue: The TGs are likely to be changed, see issue MIWP-8_(M)_Coupled_resources.
Note: there is a TODO “to validate this is the proper identification, the identification used in capabilities might be required.”
ats-metadata
A.31.IR25.resource.creation.date
Coverage: This test case only addresses the creation date (TG requirement 25). The IR require also that there will be no more than one date of revision.
ats-metadata
missing A.3X.IR24
CT critical There is no test for TG MD requirement 24, because this is claimed to be not testable.It is not clear why the temporal reference system cannot be tested.Also, the values for data of creation / revision / updated can be tested whether they conform to
Add a test case for this requirement.
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 10 of 24
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
the ISO 8601 format (yyy-mm-dd).ats-interoperability-metadata
* GE critical This ATS clearly needs more work. In general, the tests are not implementable as they are.Only IR references are provided. All tests should be against requirements stated in a TG.
ats-interoperability-metadata
A.01.IR13.1.crs
ED low Ambiguity: the hyperlink of “RS_Identifier “does not point to the right sourceIn addition “It is suggested ...” or "the test can grab ...” is not an unambiguous description of a test.Testability: Also, it is unclear how to “validate the coordinate reference system against the advertised system”?The data specifications provide http URIs for all default CRSs. Is this relevant for this test (note that http://www.opengis.net/def/crs/EPSG/0/4326 is not one of them)?
ats-interoperability-metadata
A.01.IR13.1.crs
CR medium Testability: The identifier should be checked against a code list of identifiers present in common registry (see inspire Data specification template V3.0rc3) but no official registry list is presented.
ats-interoperability-metadata
A.02.IR13.2.trs CR medium Testability: there is no code list for temporal reference systems mandated by INSPIRE.
See also MIWP-8 issue 2323.
ats-interoperability-metadata
A.03.IR13.3.enc
ED medium Ambiguity: “the format may be deduced” is too generic expression, it should either explicit or express with a code a list.
Update the INSPIRE data specifications template (Section 8.2.3) and refer to the INSPIRE media type register (See MIWP-8 issue 2324). Update the test case accordingly.
ats-interoperability-metadata
A.04.IR13.4.topo
CT, CR medium It is not clear from the TG how topological consistency is encoded.It is not clear from the test case how it can be tested.
ats-interoperability-metadata
A.04.IR13.4.topo
CT, CR medium Testability: As indicated, it is not clear how to ascertain whether ‘the data set includes types from the Generic Network Model and does not assure centreline topology (connectivity of centrelines) for the network.’
ats-interoperability-metadata
A.05.IR13.5.char.enc
CT, CR medium Testability: The test should clarify that this only works if the resource can be retrieved online and is a textual resource.Ambiguity: The test should clarify for each
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 11 of 24
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
possible resource how the character encoding of a resource can be verified. If the character encoding is not included in the retrieved resource, no claim can be made about the used character encoding. For example, in XML, the character encoding prolog is not mandatory (i.e. <?xml encoding='UTF-8'?>).
ats-discovery-service
* GE medium The test method descriptions often are not unambiguously clear with respect to exactly what needs to the asserted. Providing the Xpath references for each selection and assertion necessary would help.
ats-discovery-service
A.01.01.ISO_AP
GE critical ATS not provided as it is still a subject of further implementation specification development.
Coverage: consider of splitting into different tests; support of all mandatory operations and their parameters, correct encoding of the requests and responses
It could be possible to work with OGC to implement such a test in OGC CITE. As testing for an OGC standard is not INSPIRE-specific and should be provided by OGC.
ats-discovery-service
A.01.02.extended.behaviour
CT critical ATS not provided as it is still a subject of further implementation specification development.
Remove test case because of redundancy:“The test is an abstraction of all the tests in this Abstract Test Suite”
ats-discovery-service
A.01.03.iso_19115_19119.model
CT critical Testability + Ambiguity: both the test purpose and test method need more details and clarification in order to arrive at a test case which “is sufficient to enable a test verdict to be assigned unambiguously to each potentially observable test outcome”
Notes:Unique identification (name) for this tests needs to be added (e.g. A.1.3 Metadata request).
NR IS should be added as reference (Annex II Part A 2.1)
XPATH expression is missingats-discovery-service
A.01.04.language.parameter
ED Minor IR N2 reference should be part A and not part BXPATH expression is missing
Update reference to IR N2
Add XPATH reference
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 12 of 24
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
ats-discovery-service
A.01.05.iso-639.codes
AT medium “A.1.4 Language parameter” should be a prerequisite
ats-discovery-service
A.01.06.unsupported.languages
AT medium “A.1.4 Language parameter” should be a prerequisiteFurthermore there is some overlap with case 2 of test case A.1.4
Add 1.4 as prerequisite or consider merging with test case A.1.4
ats-discovery-service
A.02.01.iso.searching.parameters
AT medium As the list of SupportedISOQueryables and AdditionalQueryable are known an XPath could be present in the ATS.“1.3 Metadata request” should be a prerequisite
ats-discovery-service
A.02.02.additional.language.parameter
AT critical Should be merged with A.01.04
ats-discovery-service
A.02.03.addiotional.search.attributes
ED Minor Typo in the URLXPATH expression is missing
A.3.1 as prerequisite? Additional search attributes are supported by the Discovery Service.
Addiotional -> additionalAdd XPATH reference
ats-discovery-service
A.02.04.discovery.service.metadata.parameters
AT medium XPATH expression is missing
Also, clearly identify which Conformance Class / ATS needs to be passed by the document retrieved from the MetadataURL or by the ExtendedCapabilities element. For the MetadataURL it probably is the ats-metadata. For the second case I do not know any Conformance Class / ATS for this. In that case, the test cases for testing the extended capabilities would need to be specified in an unambiguous and testable way.
Add XPATH reference
ats-discovery-service
A.02.05.inspire.service
AT medium XPATH expression is missingClearly identify the Conformance Class / ATS that the response document must conform to / pass.
Add XPATH reference
ats-discovery-service
A.02.06.federated.catalogues.advertisement
CR medium Ambiguity:Neither requirement 3 nor the test states unambiguously how the list of federated catalogues shall be advertised. Adding Xpath references may help, but the information should also be included in the TG.
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 13 of 24
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
Notes:XPATH expression is missing
Under “notes”, hyperlink to section 4.3.4.3ats-discovery-service
A.02.07.federated.discovery.service
AT medium XPATH expression is missing Add XPATH reference
ats-discovery-service
A.02.08.natural.languages
AT critical It should be merged with A.01.04
ats-discovery-service
A.02.09.response.language
AT critical Could be merged with A.01.04For default language it should use “inspire_common:DefaultLanguage” as specified in A.02.10
XPath expression can be added here
Add XPATH reference
ats-discovery-service
A.02.10.supported.languages
AT medium XPATH expression is missing Add XPATH reference
ats-discovery-service
A.02.11.xml.schema
AT medium Link to the XML schema (not included in the TG) should be provided to test effectively
ats-discovery-service
A.03.01.inspire.search.attributes
ATED
medium - XPATH expression is missing- Test type can be ‘automated’- Adapt last sentence to:“This test is an abstraction of the tests A.3.7 INSPIRE search criteria, A.3.8 Language search criteria and A.3.9 Additional search criteria”
Add XPATH reference
ats-discovery-service
A.03.02.language.query.parameters
ED medium The parameter ‘Language’ could be removed from the test case as it is already present in A.01.04Reference to the TG requirement should be added. Also, it needs to be clarified what “supporting the Query parameter” means. Is it referring to test A.3.4? If yes, what does A.3.2 add?
ats-discovery-service
A.03.03.language.search.attribute
ED medium The parameter Language has been already introduced in A.01.04
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 14 of 24
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
Add A.01.04 as prerequisite or merge with A.01.04
ats-discovery-service
A.03.04.query
AT medium The XPATH expression for the query element is missing (it could be determined from the Discovery Metadata Request)
Add XPATH reference
ats-discovery-service
A.03.05.inspire
ATED
medium Testability:Not testable, as one would need to have the list of all records to verify that “each resource matching the query” is selected.
Notes:The XPATH expression is missing
URL is not descriptive, consider changing to ‘inspire.metadata’ or ‘inspire.md.elements’
Consider rephrashing purpose to make this testable: “each resource contained in the response”
Add XPATH reference
ats-discovery-service
A.03.06.distributed.search.parameter
GEED
medium Ambiguity: “Where applicable” in both test purpose and test method is vague and should be specified.The XPATH expression for hopCount is missing + information about hopCount attribute?Is the test type automated or manual?Notes section contains template text
Specify “where applicable” in both test purpose and test method
Add XPATH reference
Define whether test can be automated or not
Update notes sectionats-discovery-service
A.03.07.inspire.search.criteria
AT medium It should be merged with A.03.01 or delete A.03.01
ats-discovery-service
A.03.08.language.search.criteria
AT medium It should be merged with A.03.01 or delete A.03.01
ats-discovery-service
A.03.09.additional.search.criteria
AT medium It should be merged with A.03.01 or delete A.03.01
ats-discovery-service
A.03.10.missing.language.filter
AT critical - A.01.04 and 1.02.09 as prerequisites- I think it is in conflict with A.01.06 where a default language should be provide anyway.
A.02.09 states that if language requested by client is NOT contained in the list of supported languages, the default language
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 15 of 24
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
is used as defined in \inspire_common:Language
However based on 3.10 this should be equivalent to missing language parameter? -> In other words, the response should contain metadata records of several natural languages, as provided by the INSPIRE Discovery Service.
ats-discovery-service
A.03.11.language.filter
AT medium Overlaps with A.01.04A.01.04 as prerequisite
A.02.09 as prerequisiteats-discovery-service
A.03.12.invalid.request
AT mediumA.1.04 as prerequisite
A.2.09 as prerequisiteats-discovery-service
A.04.01.harvesting.readiness
CT medium This test (and requirement) does not belong to the ATS and the TG. It is a requirement on a different subject (some external resource), not the discovery service. Remove test from ATS. There is nothing that can be tested against the discovery service via the CSW API (or if it is clarify what the requirement is that should be tested).
Remove test
ats-discovery-service
A.04.02.third.party.discovery.services.published
CT medium Unclear what should or could be tested here via the CSW API
Clarify
ats-discovery-service
A.05.01.third.party.discovery.services.harvestable
ED medium Ambiguity: “Verification whether all information about the Public Authority’s or Third Party’s Discovery Service are provided”. The test should specify which information should be provided.
Testability: How to determine the URL of the MS discovery service? Does this need to be provided as an additional parameter to the test run?
ats-discovery-service
A.06.03.QoS.availability This is not a test for the test framework. Continuous
monitoring should be part of the local infrastructure (this applies in general for the other QoS aspects as well, but for them a one-time test in a test run are
Remove test case from ATS
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 16 of 24
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
possible, but not for availability).
ats-view-wms
* ED minor Explicit references to the implementation requirements are missing (although it can be derived from the name of the test case).
Add a reference to the implementation requirement.
ats-view-wms
IR01 CR Medium Implementation requirement 1 is too general to be tested.
As it is intended for scoping the technical guidance, it may be better to remove this as an explicit requirement from the document.
ats-view-wms
IR02, IR03 CT Medium These requirements for compliance with the “basic WMS” conformance class should perhaps be included as an explicit abstract test case.
Add this as a test case with reference to an explicit external conformance class “OGC WMS 1.3.0. Basic WMS Server”.
ats-view-wms
IR02, IR03 CT Medium The requirements related to conformance with the WMS standard are not represented in the ATS.
The ATS should reference the OGC WMS conformance class(es) that are a dependency (but not reference individual WMS tests). With a view to ETSs for those, in general, the tests for WMS 1.3.0 should be provided by OGC CITE (which would require the WMS 1.3.0 tests to be amended to be independent of any test dataset).
ats-view-wms
IR04 CR medium “The extended capabilities section shall be used to fully comply with the INSPIRE View Service metadata requirements (see section 4.2.3.3.1).”This sentence is just about scoping, as the metadata requirement is covered by IR10.
It may be better to remove this statement from the requirement.
ats-view-wms
IR06 ED medium “Mandatory ISO 19128 – WMS 1.3.0 metadata elements shall be mapped to INSPIRE metadata elements to implement a consistent interface.”This statement is redundant to IR10.
It may be better to remove this statement from IR06, as it is already covered in IR10.
ats-view-wms
IR07 ED medium “It is mandatory to use the mapping provided in this Technical Guideline (described in Section 4.2.3.3.1.1 to 4.2.3.3.1.16. INSPIRE metadata elements that cannot be mapped to available [ISO 19128] – WMS1.3.0 elements are implemented as Extended Capabilities.”This statement is redundant to IR10.
Remove this statement from IR07, as it is already covered in IR10.
ats-view-wms
IR07 ED medium “Metadata are published through a service's capabilities document and can be harvested by an INSPIRE Discovery service.”This sentence is almost literally repeated for IR09.
This sentence can be removed from IR07 as it is covered by IR09.
ats-view-wms
missing for IR09
CT medium It is claimed that this requirement is not testable. This seems wrong.
It seems that this could be testable in both scenarios:- scenario 1: Retrieve the URL to the Discovery service from <metadataURL>. Verify if the metadata record exists and is valid.
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 17 of 24
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
- scenario 2: The ATS could require an optional test run parameter to indicate the corresponding discovery service. A query could check whether there is indeed a corresponding metadata record in the discovery service.
ats-view-wms
A.02.IR04.extended.capabilities.node
ED minor “Check that the extended capabilities element validates against the INSPIRE schemas.”Coverage: it seems not correct that by applying the XSD Schema validation it is possible to validate the inspire view service metadata requirements (IR04).
There are two options:Option 1. Update IR04 removing the reference to the INSPIRE View Service metadata requirements;Option 2. Merge this test case with one or more other test cases testing the INSPIRE View Service metadata requirements (in both scenarios).
ats-view-wms
A.02.IR04.extended.capabilities.node
ED Minor Prerequisite: Test case A.03.IR05.schema.validation is a prerequisite to this test case, as it tests whether the GetCapabilities document can be retrieved and whether it is valid according to the WMS Capabilities schema.
Add A.03.IR05.schema.validation as a prerequisite.
ats-view-wms
A.03.IR05.schema.validation
CT medium Coverage: The IR05 seems to require testing in the first place whether the Capabilities document can be retrieved using valid HTTP request parameters. This requires that the GetCapabilities request should be tested without specifying a language parameter (default language) and also for all LANGUAGE parameters that the service provides.Furthermore, the test should only succeed if the Capabilities document does not contain an exception (i.e. the response can be valid against the WMS Capabilities schema when containing an exception).
First, the test should retrieve the Capabilities document using the default language (without language parameter).Different outcomes are possible:- HTTP error (e.g.4XX or 5XX) response: the test fails;- HTTP 200 response, but with service exception document: the test fails;- HTTP 200 response with capabilities document that validates against the ISO 19128 schema: the test succeeds.Second, the GetCapabilities request should be repeated for all supported languages. (>> update: this is done in test A.40)- HTTP 4XX response: the test fails;- HTTP 200 response, but with service exception document: the test fails;- HTTP 200 response with capabilities document that validates against the ISO 19128 schema: the test succeeds if the language is current.
ats-view-wms
A.04.IR06.metadataURL.node
ED medium “If no metadata URL is given then all mandatory ISO 19128 metadata elements must exist in the ExtendedCapabilities section.”It seems wrong to expect the ISO 19128 elements in the Extended element.The mandatory ISO 19128 metadata elements
The mandatory ISO 19128 elements are in the <wms:Capabilities> (<service>) elements not in the <extendedCapabilities> elmement.Remove the condition “if no metadata URL is given…”.Consider merging the check for the mandatory
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 18 of 24
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
must be provided, regardless whether or not there is a <metadataURL> link.The check for the mandatory ISO 19128 metadata elements is also covered by test cases A10, A11, A14, A19, A20, A21.
ISO 19128 metadata elements with test cases A10, A11, A14, A19, A20, A21.
ats-view-wms
A.05.IR07.extended.capabilities.elements.node
ED medium “If no metadata URL is given then all mandatory ISO 19128 metadata elements must exist in the ExtendedCapabilities section.”It seems wrong to expect the ISO 19128 elements in the Extended element.
The mandatory ISO 19128 elements are in the <wms:Capabilities> (<service>) elements not in the <extendedCapabilities> elmement.
ats-view-wms
A.05.IR07.extended.capabilities.elements.node
CT medium The reference to “ISO 19128 metadata elements“ is a 404. It is unclear what the “ISO 19128 metadata elements“ are.
Update reference to unambiguously identify the metadata elements for which test assertions are needed.
ats-view-wms
A.05.IR07.extended.capabilities.elements.node
CT medium Conditional test: This test case should only fail in case of “scenario 2” only. This means that the test case should also test for the absence of a <metadataURL> element.
It may be better to merge test cases A.04, A.05, and A.07-A.024. There are dependencies in the outcomes of these test cases.The following situations are possible:- scenario 1 only (only a <metadataURL> field): the test case should check whether all mandatory ISO19128 elements exist ánd if the referred metadata record can be retrieved and meets the INSPIRE metadata requirements;- Scenario 2 only (no <metadataURL> element): the test case should check whether all mandatory ISO19128 elements exist. Additionally, it should check whether are requirement INSPIRE metadata elements are valid;- Scenario 1 and 2 (both a <metadataURL>: the test case should check whether all mandatory ISO19128 elements exist. Additionally, it should check whether are requirement INSPIRE metadata elements are valid ánd if the referred metadata record can be retrieved and meets the INSPIRE metadata requirements.
ats-view-wms
A.06.IR08.language.node
ED minor The purpose of the test is wrong (copy-paste error). It does not refer to IR08.
Update the purpose, with the normative statement for IR08.
ats-view-wms
A.07.IR10.title.abstract
CTCR
critical Coverage: This test case does not fully tests the IR10 requirement: “An INSPIRE View service shall contain the INSPIRE metadata elements set out in the Metadata Regulation [INS MD] as shown in Table 3.”
Change the test or change the requirement only to cover keyword.It may be better to merge this test case with A.04, A05, A07, and A11-A21.
ats-view-wms
A.08.IR11.resource.type.node
CT minor This test case only needs to be executed in the context of “scenario 2”. Prerequisites should
Move assertions to the test methods. In particular, incorporate this “precondition” into the test
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 19 of 24
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
reference other test cases in the same ATS or a conformance class, not introduce additional assertions.
method. Alternatively, it may be better to merge test cases A.04, A.05, and A.07-A.024.
ats-view-wms
A.09.IR12.resource.locator.node
CT minor This test case only needs to be executed in the context of “scenario 2”. Prerequisites should reference other test cases in the same ATS or a conformance class, not introduce additional assertions.
Incorporate this “precondition” into the test method. Alternatively, it may be better to merge test cases A.04, A.05, and A.07-A.024.
ats-view-wms
A.10.IR13.coupled.resource.node
CT medium Ambiguity: “is a valid link” – how will this be unambiguously assessed? Based on the HTTP GET Response?Also, this is a duplicate of A.11.IR14Open issue: The TGs are likely to be changed, see issue MIWP-8_(M)_Coupled_resources.
Option 1: merge test case with A.10.IR13.Option 2: split the test method over the two test cases.
ats-view-wms
A.11.IR14.metadata.record.node
CT medium This test case duplicates A.10.IR13.coupled.resource.node.
Option 1: merge test case with A.10.IR13.Option 2: split the test method over the two test cases.
ats-view-wms
A.12.IR15.spatialdataservicetype.node
This test case only needs to be executed in the context of “scenario 2”. Prerequisites should reference other test cases in the same ATS or a conformance class, not introduce additional assertions. I.e. What should the test result be in case of “scenario 1” (skipped, not-applicable, etc-)?
Incorporate this “precondition” into the test method. Alternatively, it may be better to merge test cases A.04, A.05, and A.07-A.024.
ats-view-wms
IR17 CT medium Coverage: There is no test case for IR17. However, it is not clear why this requirement is not testable. Is this because the test result would always succeed, irrespective of whether or not there is a wms:KeywordList with wms:Keywords?
ats-view-wms
A.13.IR18.keywords.node
CT medium Overlap with A.39.IR16, which already checks for “/inspire_common:MandatoryKeyword”
Testability: if it is optional to define additional keywords (/inspire_common:Keyword) on top of /inspire_common:MandatoryKeyword, then this test would fail if no additional keywords are provided.
This test case only needs to be executed in the context of “scenario 2”.
Consider merging with A.39.IR16
ats-view-wms
A.15.IR20.dates.node
CT medium This test case only needs to be executed in the context of “scenario 2”. What should the test result
Rewrite the test method and provide clear instructions on the test outcome.
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 20 of 24
https://github.com/inspire-eu-validation/ats-view-wms/blob/master/A.10.IR13.coupled.resource.node.md
https://github.com/inspire-eu-validation/ats-view-wms/blob/master/A.10.IR13.coupled.resource.node.md
https://github.com/inspire-eu-validation/ats-view-wms/blob/master/A.10.IR13.coupled.resource.node.md
https://github.com/inspire-eu-validation/ats-view-wms/blob/master/A.09.IR12.resource.locator.node.md
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
be in case of “scenario 1” (skipped, not-applicable, etc-)?
ats-view-wms
A.16.IR21.temporal.reference.node
CT minor Ambiguity: This testcase as written now is already covered by test case A.15.IR20. It seems that this test case should be about the temporal extent (as a period).Also, as it is no, it is not clear under which conditions this test case fails.This test case only needs to be executed in the context of “scenario 2”.
Merge with test case A.15.IR20 and clarify in which cases (creation date, publication data, date of last revision, temporal extent) the test should fail. Please note that the combination of the requirements from the INSPIRE Metadata IR and ISO19115 make that the presence of temporal extent does not affect the outcome of the test.
NB: also in IR21, the word temporal extent should be added (as it is implied).
ats-view-wms
A.17.IR22.conformity.deegree.node
ED minor Note: in TG Requirement Coverage table (req# 22), the link to this test case returns a 404 error.This test case seems to only fail if the degree of conformity is not mentioned. Perhaps it should also fail if there is no conformity node (which is checked in A.18.IR23.conformity.node).
This test case only needs to be executed in the context of “scenario 2”.
Update link in TG Requirement Coverage table.
Consider merging this test case with A.18.IR23.conformity.node.
ats-view-wms
A.18.IR23.conformity.node
Some tags were dropped from the purpose description.
This test case only needs to be executed in the context of “scenario 2”.
Add the tags back to the purpose description.
ats-view-wms
A.19.IR24.fees.node
CT minor As ‘conditions applying to access and use’ are mandatory according to the INSPIRE Metadata IR, the test should fail when the wms:fees element is not present.
Update the test method accordingly.
ats-view-wms
A.20.IR25.contactpersonprimary.node
CT minor Currently the test only considers whether a wms:ContactPersonPrimary node exists in the wms:ContactInformation section.However, the requirement states that also a wms:ContactOrganization node should be present within wms:ContactPersonPrimary
Update test method to reflect the testing of the following structure:
/WMS_Capabilities/Service/ContactInformation
/WMS_Capabilities/Service/ContactInformation/ContactPersonPrimary
/WMS_Capabilities/Service/ContactInformation/ContactPersonPrimary/ContactOrganization
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 21 of 24
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
ats-view-wms
A.22.IR27.IR28.metadata.pointofcontact.node
CT minor This test case only needs to be executed in the context of “scenario 2”.
Incorporate this “precondition” into the test case. Alternatively, it may be better to merge test cases A.04, A.05, and A.07-A.024.
ats-view-wms
A.24.IR29.metadata.date.node
CT minor This test case only needs to be executed in the context of “scenario 2”.
Incorporate this “precondition” into the test case. Alternatively, it may be better to merge test cases A.04, A.05, and A.07-A.024.
ats-view-wms
A.26.IR31.getmap.format.node
CT minor Like for the WMTS test case, the test could also check whether the returned image is indeed encoded in the requested format.
Add: “Make a GetTile request for the layer using a maximum allowed bounding box and either "image/png" or "image/gif" format and check that the returned image is encoded in the requested format.”
ats-view-wms
IR32
ats-view-wms
A.31.IR36.layer.bbox.node
CT minor The test method does not explain how to determine “all supported CRS” to test against.
Clarify that this is based on the wms:CRS elements.
ats-view-wms
IR37 (missing test case)
CT minor It is not clear why this requirement cannot be tested.It seems that this requirement refers to the inclusion of a unique resource identifier in an external metadata record (scenario 1 – metadata record as online resource).
Clarify whether this test should be added, it could also be covered by a more general test case that retrieves the metadata record and checks whether resource identifiers are present.
ats-view-wms
A.32.IR38.layer.identifier.node
ED minor Since IR38 is split into two test cases, the test purpose should be updated to reflect this.
The test case should probably test whether all authority attributes of Identifier elements have corresponding AutorityURL elements with that name.
ats-view-wms
A.33.IR38.authority.url.node
ED medium Note: in TG Requirement Coverage table (req# 38), the link to this test case returns a 404 error.More importantly, it is not clear why there should be a AuthorityURL node in each layer. IR38 suggests that this can be defined only once. The example in the TGs also has this element only once… and has multiple layers.
Also, the test purpose is that of IR37 instead of 38.
Update the purpose. Alternatively, consider merging test cases A.32 and A.33.
ats-view-wms
A.35.IR39.harmonized.layer.name
GE minor Overlap with A.10 with regard to the testing of the HREF attribute.
ats-view-wms
A.36.IR40.etrs89.itrs.crs
GE medium Testability: refer to note in test case: “will not be complete without a machine-readable "whitelist" register of the acceptable CRSes
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 22 of 24
https://github.com/inspire-eu-validation/ats-view-wms/blob/master/A.35.IR39.harmonized.layer.name.md
https://github.com/inspire-eu-validation/ats-view-wms/blob/master/A.35.IR39.harmonized.layer.name.md
https://github.com/inspire-eu-validation/ats-view-wms/blob/master/A.35.IR39.harmonized.layer.name.md
https://github.com/inspire-eu-validation/ats-view-wms/blob/master/A.32.IR38.layer.identifier.node.md
https://github.com/inspire-eu-validation/ats-view-wms/blob/master/A.32.IR38.layer.identifier.node.md
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
with their CRS identifiers and commonly used aliases”
ats-view-wms
IR49 (missing) CT Medium It is not clear why IR49 on category layer MetadataURL cannot be tested.
ats-view-wms
IR50-IR59 CT Medium These requirements for conformance with the WMS standard should perhaps be included as explicit abstract test cases that make reference to the corresponding OGC ATSs.
Add explicit abstract test cases for these requirements with reference to the corresponding OGC ATSs.
ats-view-wms
A.39.IR16.spatial.data.service.keyword.embedded.metadata
Test case already includes a test for this.This test case only needs to be executed in the context of “scenario 2”.
Consider merging with A.13.IR18.keywords.node.
ats-view-wmts
IR74-IR75, IR78, IR81, IR82
CT Medium The requirements related to conformance with the WMTS standard are not represented in the ATS.
The ATS should reference the OGC WMTS conformance class(es) that are a dependency (but not reference individual WMTS tests). With a view to ETSs for those, in general, the tests for WMTS 1.0.0 should be provided by OGC CITE.
ats-view-wmts
* ED minor Explicit references to the implementation requirements are missing (although it can be sometimes derived from the name of the test case).
Add a reference to the implementation requirement.
ats-view-wmts *
GE medium Prerequisites should reference other test cases in the same ATS or a conformance class, not introduce additional assertions.
Move assertions to the test methods.
ats-view-wmts *
GE medium Link to specific test case is missing for prerequisites
Update prerequisites with specific reference tests
ats-view-wmts
A.01.IR77.language.param
ED medium The meaning of “RESTful or procedure oriented” could be clarified.
ats-view-wmts A.02.IR79.lay
er.metadata.ref
GEED
medium Ambiguity: What is “a valid WMTS 1.0.0 ServiceMetadata document”? Is it a WMTS 1.0.0 Capabilities document that is schema valid?Notes: First sentence in “purpose” section: a unambiguous -> an unambiguous
ats-view-wmts
A.03.IR82.image.format
ED medium Reference refers to Chapter 5.2.3.3.2.2, while the correct chapter would be Chapter 4.2.3.3.2.2
Update reference to TG VS Chapter 4.2.3.3.2.2
ats-view-wmts
A.04.layer.name.id
ED medium Ambiguity: Wording of purpose not entirely clear: “It must be unambiguous to find out which of the layers provided by the service visualize the INSPIRE spatial data sets given in the Data Specifications for each INSPIRE theme”Testability: “the list of valid layer names should
Clarify that the correct harmonised layer names (which are codes ) are indeed in the inspire registry: http://inspire.ec.europa.eu/layer
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 23 of 24
Template for comments Reviewer: ARE3NA ISA action Date: 2016-03-11 Document: Draft Abstract Test Suites Project: MIWP-5 / MIG-T
ATS(e.g.
download-atom)
Test(e.g.
A.01.TGR1)
Type of comment1
Severity(minor,
medium, critical)
Comments Proposed change Resolution
be in the INSPIRE registry, if it is supposed to be used in a test.Notes:Open questions/notes on the usefulness of harmonised layer names
ats-view-wmts
A.05.IR85.layer.title
ED minor In purpose: users’ -> usersWill validation of the correctness of the translated title be included (see note)?
Evaluate and update test method.Clarify whether the translations need to be tested, and whether the correct harmonised layer names (which are codes) and the translated labels to be used are in the inspire registry: http://inspire.ec.europa.eu/layer
ats-view-wmts
A.05.IR85.layer.title
ED Minor Add an explicit reference to implementation requirement 85
Add an explicit reference to implementation requirement 85
ats-view-wmts A.06.IR86.lay
er.abstract
AT critical Test method only verifies whether abstract is a non-empty character string. There is no check to validate that the language presented is the correct one.
Evaluate and update test method
ats-view-wmts A.07.IR88.lay
er.bboxAT critical The test method states “longitude and latitude, in
this order” – How to test that a value is longitude or latitude?
Evaluate and update test method
ats-view-wmts A.08.IR90.lay
er.style
AT critical Test purpose states that at least one style element exists per layer. However, the test method does not include a check on the existence of at least one layer style.
Update test method to include check on whether at least one layer style exists.
ats-view-wmts A.09.IR91.lay
er.legend
CR critical It is not possible to check whether the resolved legend resource is a valid image as the format attribute has no requirements in the TG (see notes).
Evaluate and update test method
1 Type of comment: GE = general ED = editorial AT = add a test (that is currently missing to test a certain requirement) CT = we have a different interpretation of how the TG requirement should be tested (and therefore propose to change the test)CR = we have a different interpretation of how the IR requirement has been translated into a TG requirement (and therefore propose to change the requirement)
page 24 of 24
