dda-wizard issueshttps://git.gesis.org/dda/dda-wizard/-/issues2017-05-18T11:17:27Zhttps://git.gesis.org/dda/dda-wizard/-/issues/54Fix OAI-PMH *from* race condition between server and FeederService2017-05-18T11:17:27ZGerrit HübbersFix OAI-PMH *from* race condition between server and FeederServiceAn OAI-PMH server may update its data only once per day during a cronjob.
FeederService.feed(..) and BundlesSourceService.updateAfterFeeding(..) currently assume, with respect with their update to the incremental harvesting "from" param...An OAI-PMH server may update its data only once per day during a cronjob.
FeederService.feed(..) and BundlesSourceService.updateAfterFeeding(..) currently assume, with respect with their update to the incremental harvesting "from" parameter that the OAI-PMH server data is up-to-date, serving out all OAI records between `from` and now. Here lies a race condition, exmplified with the following scenario.
todayAt22hrs: SchedulerConfiguration triggers -> FeederService harvests from lastDayAt22hrs until now (i.e. ~ todayAt22hrs) ... the OAI-PMH server does not report any new records **in this timeframe**. therefore, FeederServer doesn't get returned any records. FeederServer updates the `from` value to `today at 22 hrs`.
todayAt23hrs: the daily OAI-PMH server cronjob kicks in and adds to its index **all new records that have appeared between lastDayAt23hrs until now (i.e. ~ todayAt23hrs). It adds new records with their `lastModifiedDate` value set to the corresponding time when these records were added during the OAI-PMH server editor's business hours.
This repeats on and on with no records ever being added, and actually records being skipped.
The Solution would be to have `from` not being updated to the time of startOfTodaysIncrementalHarvesting, but instead keep (or update) it to the value of the last latest identified `lastModifiedDate` encountered in some previous harvesting ... this may result in the same from interval start being queried continously for several days until at least one next record is found - whose lastModifiedDate will be used.Gerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/53Sort source, transformed, and target repository metadata on some bundle's det...2019-01-23T10:59:15ZNawaz, Muhammad AhsanSort source, transformed, and target repository metadata on some bundle's detail page nicelyI'd be nice to have the three different sets of metadata associated with a bundle, with identical keys on one line.
Example:
```
key || source metadata | transformed metadata | target repository metadata
----...I'd be nice to have the three different sets of metadata associated with a bundle, with identical keys on one line.
Example:
```
key || source metadata | transformed metadata | target repository metadata
-------------------------||-----------------|----------------------|---------------------------
dc.title || HELLO | Hello | Hello world
bla.uninteresting || dkfjdkj | |
internal.dda.identifier || | somewhere@@foo | somewhere@@foo
ssoar.identifier.classoz || | | 20059
```https://git.gesis.org/dda/dda-wizard/-/issues/50Use new BundleService in FeederService2017-05-18T11:17:27ZGerrit HübbersUse new BundleService in FeederServiceGerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/48Make FeederService transactionally more granular2017-05-18T11:17:27ZGerrit HübbersMake FeederService transactionally more granularRight now, FeederService will commit found and updated Bundles and BundlesSource only after it has finished feeding.
In case something breaks during feeding (e.g. restart), all state that has been changed up until then will be lost.
In...Right now, FeederService will commit found and updated Bundles and BundlesSource only after it has finished feeding.
In case something breaks during feeding (e.g. restart), all state that has been changed up until then will be lost.
Instead, the transactions should commit for each processed bundle.Gerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/46Use in Developer Profil a Local MySql Database Instead of H22018-02-07T16:35:52ZHellmich, ChristophUse in Developer Profil a Local MySql Database Instead of H2Fehler die nur in Verbindung mit einer MySQL-Datenbank auftreten, sind aufwändig zu beheben, da der Fehler nicht lokal untersucht werden kann.Fehler die nur in Verbindung mit einer MySQL-Datenbank auftreten, sind aufwändig zu beheben, da der Fehler nicht lokal untersucht werden kann.https://git.gesis.org/dda/dda-wizard/-/issues/45Rename Entity Options to Option2019-01-23T11:00:33ZHellmich, ChristophRename Entity Options to OptionUmbenennung der Entität von Plural in Singular.
Hinweis: Erst ab Version 3 von JHipster können Entitäten erstellen werden, deren Namen einem *reserved Word* des DBMSs entsprechen [1]. Da zurzeit Version 2.27.2 von JHipster verwendet wir...Umbenennung der Entität von Plural in Singular.
Hinweis: Erst ab Version 3 von JHipster können Entitäten erstellen werden, deren Namen einem *reserved Word* des DBMSs entsprechen [1]. Da zurzeit Version 2.27.2 von JHipster verwendet wird, ist eine automatische Generierung der Entität einschließlich aller weiteren Dateien nicht möglich.
[1] https://github.com/jhipster/generator-jhipster/issues/2558https://git.gesis.org/dda/dda-wizard/-/issues/44Update main site from normal user2019-01-23T11:00:33ZHellmich, ChristophUpdate main site from normal userhttps://git.gesis.org/dda/dda-wizard/-/issues/43Add setSpec data to BundlesSource entity2020-02-28T08:59:53ZGerrit HübbersAdd setSpec data to BundlesSource entitySome OAI-PMH-kind BundlesSources require to have their search specification scoped on a specific set, e.g. for harvesting the discussion papers sets from WZB.Some OAI-PMH-kind BundlesSources require to have their search specification scoped on a specific set, e.g. for harvesting the discussion papers sets from WZB.Gerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/42introduce metadatum for time of harvest to help with debugging2017-05-18T11:17:27ZGerrit Hübbersintroduce metadatum for time of harvest to help with debugginge.g. internal.dda.harvestdatee.g. internal.dda.harvestdatehttps://git.gesis.org/dda/dda-wizard/-/issues/40create sample Bundle data2017-05-18T11:17:27ZGerrit Hübberscreate sample Bundle dataGerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/38Set up button-press deployment to dda.ssoar.info2017-05-18T11:17:27ZGerrit HübbersSet up button-press deployment to dda.ssoar.infoSimilar to what was done for www.ssoar.info .Similar to what was done for www.ssoar.info .Gerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/37WbvResolvingStrategy should retry seed URL2017-05-18T11:17:27ZGerrit HübbersWbvResolvingStrategy should retry seed URLStacktrace:
```
main] o.g.d.p.domain.impl.ContentResolver : seedUrlString is https://www.wbv.de/journals/zeitschriften/die-zeitschrift-fuer-erwachsenenbildung/artikel/shop/detail/name/_/0/1/DIE1301W043/facet/DIE1301W043////...Stacktrace:
```
main] o.g.d.p.domain.impl.ContentResolver : seedUrlString is https://www.wbv.de/journals/zeitschriften/die-zeitschrift-fuer-erwachsenenbildung/artikel/shop/detail/name/_/0/1/DIE1301W043/facet/DIE1301W043///////nb/0/category/730.html
2017-03-08 19:13:18.525 WARN 431948 --- [ main] o.g.d.p.d.impl.WbvResolvingStrategy : Could not find InputStream for seed URL https://www.wbv.de/journals/zeitschriften/die-zeitschrift-fuer-erwachsenenbildung/artikel/shop/detail/name/_/0/1/DIE1301W043/facet/DIE1301W043///////nb/0/category/730.html
java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:792) ~[na:1.8.0_112]
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647) ~[na:1.8.0_112]
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:789) ~[na:1.8.0_112]
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647) ~[na:1.8.0_112]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569) ~[na:1.8.0_112]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474) ~[na:1.8.0_112]
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) ~[na:1.8.0_112]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338) ~[na:1.8.0_112]
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:567) ~[jsoup-1.9.2.jar:na]
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:540) ~[jsoup-1.9.2.jar:na]
at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:227) ~[jsoup-1.9.2.jar:na]
at org.jsoup.helper.HttpConnection.get(HttpConnection.java:216) ~[jsoup-1.9.2.jar:na]
at org.gesis.dda.publishing.domain.impl.WbvResolvingStrategy.getContentsUrls(WbvResolvingStrategy.java:29) ~[classes/:na]
at org.gesis.dda.publishing.domain.impl.ContentResolver.resolveLinks(ContentResolver.java:305) [classes/:na]
at org.gesis.dda.publishing.domain.impl.ContentResolver.resolveContentsUrls(ContentResolver.java:225) [classes/:na]
at org.gesis.dda.publishing.domain.impl.ContentResolver.resolveContentsSuppliers(ContentResolver.java:331) [classes/:na]
at org.gesis.dda.publishing.domain.impl.AutonomouslyContentResolvingBundle.getContentsSuppliers(AutonomouslyContentResolvingBundle.java:41) [classes/:na]
at org.gesis.dda.publishing.domain.impl.SsoarXmlWorkflowRepository.addBundle(SsoarXmlWorkflowRepository.java:82) [classes/:na]
at org.gesis.dda.publishing.domain.impl.SsoarXmlWorkflowRepository.addBundle(SsoarXmlWorkflowRepository.java:25) [classes/:na]
at org.gesis.dda.wizard.service.FeederService.lambda$0(FeederService.java:96) [classes/:na]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_112]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_112]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_112]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_112]
at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[na:1.8.0_112]
at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[na:1.8.0_112]
at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:743) ~[na:1.8.0_112]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_112]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_112]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[na:1.8.0_112]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_112]
at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:474) ~[na:1.8.0_112]
at org.gesis.dda.wizard.service.FeederService.feed(FeederService.java:112) [classes/:na]
at org.gesis.dda.wizard.service.FeederService$$FastClassBySpringCGLIB$$d6ecb794.invoke(<generated>) ~[classes/:na]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720) ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) ~[spring-tx-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281) ~[spring-tx-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655) ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.gesis.dda.wizard.service.FeederService$$EnhancerBySpringCGLIB$$d59945ed.feed(<generated>) ~[classes/:na]
at org.gesis.dda.wizard.service.FeederServiceIntTest.feedOaiPmh(FeederServiceIntTest.java:94) ~[test-classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_112]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_112]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_112]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_112]
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) ~[junit-4.12.jar:4.12]
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) ~[junit-4.12.jar:4.12]
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) ~[junit-4.12.jar:4.12]
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) ~[junit-4.12.jar:4.12]
at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75) ~[spring-test-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86) ~[spring-test-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84) ~[spring-test-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) ~[junit-4.12.jar:4.12]
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:254) ~[spring-test-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:89) ~[spring-test-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) ~[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) ~[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) ~[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) ~[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) ~[junit-4.12.jar:4.12]
at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61) ~[spring-test-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70) ~[spring-test-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.junit.runners.ParentRunner.run(ParentRunner.java:363) ~[junit-4.12.jar:4.12]
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:193) ~[spring-test-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86) ~[.cp/:na]
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) ~[.cp/:na]
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459) ~[.cp/:na]
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:678) ~[.cp/:na]
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382) ~[.cp/:na]
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192) ~[.cp/:na]
```Gerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/36Set up AngularJS overview to show all Bundles for a selected BundlesSource2019-01-23T11:00:34ZGerrit HübbersSet up AngularJS overview to show all Bundles for a selected BundlesSourceRelevant files for the front-end:
src/main/webapp/scripts/app/entities/bundlesSource/bundlesSource-detail.html
src/main/webapp/components/entities/bundlesSource/bundlesSource.service.js
A problem exists with the REST endpoint n...Relevant files for the front-end:
src/main/webapp/scripts/app/entities/bundlesSource/bundlesSource-detail.html
src/main/webapp/components/entities/bundlesSource/bundlesSource.service.js
A problem exists with the REST endpoint not returning some bundlesSource's "bundles". @christoph.hellmich ran into a similar challenges with Wizard entity's "steps" relationship. The solution found back then was
* to remove the @JsonIgnore annotation from Wizard entity's "steps" field and fetch eagerly
* and introducing the @JsonView(xyz) annotation on entity fields and introducing the @JsonView(xyz) annotation on WizardResource's REST methods, e.g. `org.gesis.dda.wizard.web.rest.WizardResource.getAllWizards()`
---
Gerrit will update file `bundlesSource.service.js` to make the AngularJS side understand bundlesSource's "bundles" relationship. http://stackoverflow.com/a/14530508/923560 could be a possible solution.https://git.gesis.org/dda/dda-wizard/-/issues/35investigate PICA format from DSM2018-01-11T10:56:17ZGerrit Hübbersinvestigate PICA format from DSMGerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/34harvest ILSHS2017-08-15T11:17:08ZGerrit Hübbersharvest ILSHSGerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/33remove default DDA user password for production environment2017-05-18T11:17:28ZGerrit Hübbersremove default DDA user password for production environmentFind a clever solution so that any production installation's default users has a randomly generated password.
default users:
* admin
* developer
* user
* ?
add passage to documentation.Find a clever solution so that any production installation's default users has a randomly generated password.
default users:
* admin
* developer
* user
* ?
add passage to documentation.https://git.gesis.org/dda/dda-wizard/-/issues/32Set up production machine reverse proxy for dda.ssoar.info2017-05-18T11:17:28ZGerrit HübbersSet up production machine reverse proxy for dda.ssoar.infoalso, put what was learned into README.MD documentation.also, put what was learned into README.MD documentation.Gerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/31Set up production environment e-mail sending2017-05-18T11:17:28ZGerrit HübbersSet up production environment e-mail sendingsudo service dda-wizard start
http://87.106.249.160:8081/sudo service dda-wizard start
http://87.106.249.160:8081/Gerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/29Add metadata to Bundles entity2018-01-11T10:59:38ZGerrit HübbersAdd metadata to Bundles entity* shall it be either transformed/harmonized metadata or original/raw metadata? Or both?
* shall it include bitstreams or not? (probably no bitstream, as this is lots of filesystem data for DDA broker server)* shall it be either transformed/harmonized metadata or original/raw metadata? Or both?
* shall it include bitstreams or not? (probably no bitstream, as this is lots of filesystem data for DDA broker server)Gerrit HübbersGerrit Hübbershttps://git.gesis.org/dda/dda-wizard/-/issues/28Introduce OAI-PMH incremental harvesting concept2017-05-18T11:17:28ZGerrit HübbersIntroduce OAI-PMH incremental harvesting concept* iff harvesting is successful, update BundlesSource data with timestamp of that successful harvesting's starting time
* introduce scheduler (once per day) to trigger harvesting
* what about target-repository-existing records for which u...* iff harvesting is successful, update BundlesSource data with timestamp of that successful harvesting's starting time
* introduce scheduler (once per day) to trigger harvesting
* what about target-repository-existing records for which updated metadata was received via OAI-PMH? Update metadata without editor intervention? create a new version of that target repository item?Gerrit HübbersGerrit Hübbers