Only some documents arrive in gogs/das-ddi32 after big change in gogs/dsdm-ddi32
Description
- Push of big commit to gogs/dsdm-ddi32 triggers the merge of study and variable descriptions
- But in gogs/das-ddi32 only a subset of documents arrives (see below)
- No logs of
converter
runtime onsvko-docker2
: Configuration has been changed in repo, but forgotten to be re-deployed😭
Possible reason
- Until now, the full run of transformations is not made inside a docker container, only without docker on machines with enough memory
- Some transformations executed in parallel have failed because of memory limits of the container
Todos
-
Execute transformations not in parallel -
Log duration of each transformation -
Ensure that logs of converter
service are mapped tosvko-docker2
-
Provide portainer to view the converter
logs for feedback- Login with gesis credentials
- To see stack user must be assigned by admin @alexander.muehlbauer
-
Short-term trigger of full transformation so that gogs/das-ddi32 is up-to-date. -
Trigger full import in http://das-ddi32.flatdb.svko-exploredata-staging.gesis.intra -
Provide new flatdb data image of latest gogs/das-ddi32
Ideas
- Provide how to trigger a full transformation
- Provide property to switch off parallel xslt transformation execution
- Provide web endpoint to trigger transformations
Resources
Edited by Alexander Mühlbauer