dda-wizard issueshttps://git.gesis.org/dda/dda-wizard/-/issues2018-01-11T11:06:23Zhttps://git.gesis.org/dda/dda-wizard/-/issues/56Create a SSOAR-in-a-box for integration test purposes2018-01-11T11:06:23ZGerrit HübbersCreate a SSOAR-in-a-box for integration test purposesGerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/55Rewrite KnownMetadataPrefixesOaiPmhBundlesStreamSource to just use Instant fo...2017-05-18T11:17:27ZGerrit HübbersRewrite KnownMetadataPrefixesOaiPmhBundlesStreamSource to just use Instant for `from` and `until`, even for day-granularity OAI-PMH serversThe OAI-PMH 2 specification says:
```
Dates and times are uniformly encoded using ISO8601 and are expressed in UTC throughout the protocol. When time is included, the special UTC designator ("Z") must be used. UTC is implied for dates al...The OAI-PMH 2 specification says:
```
Dates and times are uniformly encoded using ISO8601 and are expressed in UTC throughout the protocol. When time is included, the special UTC designator ("Z") must be used. UTC is implied for dates although no timezone designator is specified. For example, 1957-03-20T20:30:00Z is UTC 8:30:00 PM on March 20th 1957. UTCdatetime is used in both protocol requests and protocol replies, in the way described in the following sections.
```
As the day-granularity type is supposed to use UTC / Zulu time, convertions from an internal `Instant` representation should be correct with respect to the specification.
The rewrite shall include changes to `org.gesis.dda.harvester.oaipmh.impl.HarvestSpecificationBuilder`, introducing a method `public void setIntervalTpye(HarvestingIntervalType intervalType)`. That new method in turn can then be programmatically used in `KnownMetadataPrefixesOaiPmhBundlesStreamSource` by querying oaiClient.identify().getGranularity().
As a result, using `KnownMetadataPrefixesOaiPmhBundlesStreamSource` shall become easier, as its clients no longer have the burden of deciding for the granularity type and by implication abstracting away *if-then-elsing* between second-granularity Instant and day-granularity LocalDate.
When this rewrite is finished, it is a good time to extract `KnownMetadataPrefixesOaiPmhBundlesStreamSource` into the YOAI library.Gerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/54Fix OAI-PMH *from* race condition between server and FeederService2017-05-18T11:17:27ZGerrit HübbersFix OAI-PMH *from* race condition between server and FeederServiceAn OAI-PMH server may update its data only once per day during a cronjob.
FeederService.feed(..) and BundlesSourceService.updateAfterFeeding(..) currently assume, with respect with their update to the incremental harvesting "from" param...An OAI-PMH server may update its data only once per day during a cronjob.
FeederService.feed(..) and BundlesSourceService.updateAfterFeeding(..) currently assume, with respect with their update to the incremental harvesting "from" parameter that the OAI-PMH server data is up-to-date, serving out all OAI records between `from` and now. Here lies a race condition, exmplified with the following scenario.
todayAt22hrs: SchedulerConfiguration triggers -> FeederService harvests from lastDayAt22hrs until now (i.e. ~ todayAt22hrs) ... the OAI-PMH server does not report any new records **in this timeframe**. therefore, FeederServer doesn't get returned any records. FeederServer updates the `from` value to `today at 22 hrs`.
todayAt23hrs: the daily OAI-PMH server cronjob kicks in and adds to its index **all new records that have appeared between lastDayAt23hrs until now (i.e. ~ todayAt23hrs). It adds new records with their `lastModifiedDate` value set to the corresponding time when these records were added during the OAI-PMH server editor's business hours.
This repeats on and on with no records ever being added, and actually records being skipped.
The Solution would be to have `from` not being updated to the time of startOfTodaysIncrementalHarvesting, but instead keep (or update) it to the value of the last latest identified `lastModifiedDate` encountered in some previous harvesting ... this may result in the same from interval start being queried continously for several days until at least one next record is found - whose lastModifiedDate will be used.Gerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/53Sort source, transformed, and target repository metadata on some bundle's det...2019-01-23T10:59:15ZNawaz, Muhammad AhsanSort source, transformed, and target repository metadata on some bundle's detail page nicelyI'd be nice to have the three different sets of metadata associated with a bundle, with identical keys on one line.
Example:
```
key || source metadata | transformed metadata | target repository metadata
----...I'd be nice to have the three different sets of metadata associated with a bundle, with identical keys on one line.
Example:
```
key || source metadata | transformed metadata | target repository metadata
-------------------------||-----------------|----------------------|---------------------------
dc.title || HELLO | Hello | Hello world
bla.uninteresting || dkfjdkj | |
internal.dda.identifier || | somewhere@@foo | somewhere@@foo
ssoar.identifier.classoz || | | 20059
```https://git.gesis.org/dda/dda-wizard/-/issues/52Think about changing entities Setting, ElementOption, Validation to value obj...2018-01-11T11:08:22ZGerrit HübbersThink about changing entities Setting, ElementOption, Validation to value objects/@EmbeddableCompare to how it was done for the @Embeddable JpaMetadatum used in the @Entity Bundle.Compare to how it was done for the @Embeddable JpaMetadatum used in the @Entity Bundle.https://git.gesis.org/dda/dda-wizard/-/issues/51Replace com.github.ladutsko.isbn** packages with Maven dependency2018-01-11T11:08:22ZGerrit HübbersReplace com.github.ladutsko.isbn** packages with Maven dependencyvia https://github.com/jitpack/jitpack.io/issues/1107via https://github.com/jitpack/jitpack.io/issues/1107Gerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/50Use new BundleService in FeederService2017-05-18T11:17:27ZGerrit HübbersUse new BundleService in FeederServiceGerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/49E-mails sent by DDA-uploaded SSOAR documents sould not be empty2018-01-11T11:06:19ZGerrit HübbersE-mails sent by DDA-uploaded SSOAR documents sould not be emptyvia Thomas:
```
Hallo Gerrit,
bislang sind die E-Mails, die uns darüber informieren, dass neue Dokumente hochgeladen werden, für den DDA noch leer (s.u.). Für zukünftige Import ist es sinnvoll, wenn auch das Titelfeld befüllt ist.
Viel...via Thomas:
```
Hallo Gerrit,
bislang sind die E-Mails, die uns darüber informieren, dass neue Dokumente hochgeladen werden, für den DDA noch leer (s.u.). Für zukünftige Import ist es sinnvoll, wenn auch das Titelfeld befüllt ist.
Viele Grüße
Thomas
```
* [ ] Task von UweGerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/48Make FeederService transactionally more granular2017-05-18T11:17:27ZGerrit HübbersMake FeederService transactionally more granularRight now, FeederService will commit found and updated Bundles and BundlesSource only after it has finished feeding.
In case something breaks during feeding (e.g. restart), all state that has been changed up until then will be lost.
In...Right now, FeederService will commit found and updated Bundles and BundlesSource only after it has finished feeding.
In case something breaks during feeding (e.g. restart), all state that has been changed up until then will be lost.
Instead, the transactions should commit for each processed bundle.Gerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/47Treat deleted OAI-PMH records well2018-01-11T11:07:13ZGerrit HübbersTreat deleted OAI-PMH records wellTreat deleted OAI-PMH records correctly, e.g.
http://dspace.wbv.de:8888/oai/request?verb=ListIdentifiers&metadataPrefix=xoai
lists an identifier `oai:dspace.wbv.de:103278/523`, but querying for that identifier
http://dspace.wbv.de:8888/...Treat deleted OAI-PMH records correctly, e.g.
http://dspace.wbv.de:8888/oai/request?verb=ListIdentifiers&metadataPrefix=xoai
lists an identifier `oai:dspace.wbv.de:103278/523`, but querying for that identifier
http://dspace.wbv.de:8888/oai/request?verb=GetRecord&metadataPrefix=xoai&identifier=oai:dspace.wbv.de:103278/523
results in a *The given id does not exist* error message.
DDA should filter out such records, possibly marking them as DELETED in the BundleEntityGerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/46Use in Developer Profil a Local MySql Database Instead of H22018-02-07T16:35:52ZHellmich, ChristophUse in Developer Profil a Local MySql Database Instead of H2Fehler die nur in Verbindung mit einer MySQL-Datenbank auftreten, sind aufwändig zu beheben, da der Fehler nicht lokal untersucht werden kann.Fehler die nur in Verbindung mit einer MySQL-Datenbank auftreten, sind aufwändig zu beheben, da der Fehler nicht lokal untersucht werden kann.https://git.gesis.org/dda/dda-wizard/-/issues/45Rename Entity Options to Option2019-01-23T11:00:33ZHellmich, ChristophRename Entity Options to OptionUmbenennung der Entität von Plural in Singular.
Hinweis: Erst ab Version 3 von JHipster können Entitäten erstellen werden, deren Namen einem *reserved Word* des DBMSs entsprechen [1]. Da zurzeit Version 2.27.2 von JHipster verwendet wir...Umbenennung der Entität von Plural in Singular.
Hinweis: Erst ab Version 3 von JHipster können Entitäten erstellen werden, deren Namen einem *reserved Word* des DBMSs entsprechen [1]. Da zurzeit Version 2.27.2 von JHipster verwendet wird, ist eine automatische Generierung der Entität einschließlich aller weiteren Dateien nicht möglich.
[1] https://github.com/jhipster/generator-jhipster/issues/2558https://git.gesis.org/dda/dda-wizard/-/issues/44Update main site from normal user2019-01-23T11:00:33ZHellmich, ChristophUpdate main site from normal userhttps://git.gesis.org/dda/dda-wizard/-/issues/43Add setSpec data to BundlesSource entity2020-02-28T08:59:53ZGerrit HübbersAdd setSpec data to BundlesSource entitySome OAI-PMH-kind BundlesSources require to have their search specification scoped on a specific set, e.g. for harvesting the discussion papers sets from WZB.Some OAI-PMH-kind BundlesSources require to have their search specification scoped on a specific set, e.g. for harvesting the discussion papers sets from WZB.Gerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/42introduce metadatum for time of harvest to help with debugging2017-05-18T11:17:27ZGerrit Hübbersintroduce metadatum for time of harvest to help with debugginge.g. internal.dda.harvestdatee.g. internal.dda.harvestdatehttps://git.gesis.org/dda/dda-wizard/-/issues/41Write D-Lib Magazine paper for dublettencheck2018-01-11T11:07:26ZGerrit HübbersWrite D-Lib Magazine paper for dublettencheckvia Philipp Mayrvia Philipp MayrGerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/40create sample Bundle data2017-05-18T11:17:27ZGerrit Hübberscreate sample Bundle dataGerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/39Analyze PDFs for interesting metadata2018-01-11T11:07:51ZGerrit HübbersAnalyze PDFs for interesting metadataGerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/38Set up button-press deployment to dda.ssoar.info2017-05-18T11:17:27ZGerrit HübbersSet up button-press deployment to dda.ssoar.infoSimilar to what was done for www.ssoar.info .Similar to what was done for www.ssoar.info .Gerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/37WbvResolvingStrategy should retry seed URL2017-05-18T11:17:27ZGerrit HübbersWbvResolvingStrategy should retry seed URLStacktrace:
```
main] o.g.d.p.domain.impl.ContentResolver : seedUrlString is https://www.wbv.de/journals/zeitschriften/die-zeitschrift-fuer-erwachsenenbildung/artikel/shop/detail/name/_/0/1/DIE1301W043/facet/DIE1301W043////...Stacktrace:
```
main] o.g.d.p.domain.impl.ContentResolver : seedUrlString is https://www.wbv.de/journals/zeitschriften/die-zeitschrift-fuer-erwachsenenbildung/artikel/shop/detail/name/_/0/1/DIE1301W043/facet/DIE1301W043///////nb/0/category/730.html
2017-03-08 19:13:18.525 WARN 431948 --- [ main] o.g.d.p.d.impl.WbvResolvingStrategy : Could not find InputStream for seed URL https://www.wbv.de/journals/zeitschriften/die-zeitschrift-fuer-erwachsenenbildung/artikel/shop/detail/name/_/0/1/DIE1301W043/facet/DIE1301W043///////nb/0/category/730.html
java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:792) ~[na:1.8.0_112]
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647) ~[na:1.8.0_112]
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:789) ~[na:1.8.0_112]
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647) ~[na:1.8.0_112]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569) ~[na:1.8.0_112]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474) ~[na:1.8.0_112]
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) ~[na:1.8.0_112]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338) ~[na:1.8.0_112]
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:567) ~[jsoup-1.9.2.jar:na]
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:540) ~[jsoup-1.9.2.jar:na]
at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:227) ~[jsoup-1.9.2.jar:na]
at org.jsoup.helper.HttpConnection.get(HttpConnection.java:216) ~[jsoup-1.9.2.jar:na]
at org.gesis.dda.publishing.domain.impl.WbvResolvingStrategy.getContentsUrls(WbvResolvingStrategy.java:29) ~[classes/:na]
at org.gesis.dda.publishing.domain.impl.ContentResolver.resolveLinks(ContentResolver.java:305) [classes/:na]
at org.gesis.dda.publishing.domain.impl.ContentResolver.resolveContentsUrls(ContentResolver.java:225) [classes/:na]
at org.gesis.dda.publishing.domain.impl.ContentResolver.resolveContentsSuppliers(ContentResolver.java:331) [classes/:na]
at org.gesis.dda.publishing.domain.impl.AutonomouslyContentResolvingBundle.getContentsSuppliers(AutonomouslyContentResolvingBundle.java:41) [classes/:na]
at org.gesis.dda.publishing.domain.impl.SsoarXmlWorkflowRepository.addBundle(SsoarXmlWorkflowRepository.java:82) [classes/:na]
at org.gesis.dda.publishing.domain.impl.SsoarXmlWorkflowRepository.addBundle(SsoarXmlWorkflowRepository.java:25) [classes/:na]
at org.gesis.dda.wizard.service.FeederService.lambda$0(FeederService.java:96) [classes/:na]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_112]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_112]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_112]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_112]
at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[na:1.8.0_112]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[na:1.8.0_112]
at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:743) ~[na:1.8.0_112]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_112]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_112]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[na:1.8.0_112]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_112]
at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:474) ~[na:1.8.0_112]
at org.gesis.dda.wizard.service.FeederService.feed(FeederService.java:112) [classes/:na]
at org.gesis.dda.wizard.service.FeederService$$FastClassBySpringCGLIB$$d6ecb794.invoke(<generated>) ~[classes/:na]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720) ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) ~[spring-tx-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281) ~[spring-tx-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655) ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.gesis.dda.wizard.service.FeederService$$EnhancerBySpringCGLIB$$d59945ed.feed(<generated>) ~[classes/:na]
at org.gesis.dda.wizard.service.FeederServiceIntTest.feedOaiPmh(FeederServiceIntTest.java:94) ~[test-classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_112]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_112]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_112]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_112]
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) ~[junit-4.12.jar:4.12]
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) ~[junit-4.12.jar:4.12]
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) ~[junit-4.12.jar:4.12]
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) ~[junit-4.12.jar:4.12]
at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75) ~[spring-test-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86) ~[spring-test-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84) ~[spring-test-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) ~[junit-4.12.jar:4.12]
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:254) ~[spring-test-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:89) ~[spring-test-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) ~[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) ~[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) ~[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) ~[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) ~[junit-4.12.jar:4.12]
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61) ~[spring-test-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70) ~[spring-test-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.junit.runners.ParentRunner.run(ParentRunner.java:363) ~[junit-4.12.jar:4.12]
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:193) ~[spring-test-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86) ~[.cp/:na]
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) ~[.cp/:na]
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459) ~[.cp/:na]
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:678) ~[.cp/:na]
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382) ~[.cp/:na]
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192) ~[.cp/:na]
```Gerrit HübbersGerrit Hübbers