How to test a new Docker image for digdag workflow on CircleCI?
Testing workflow runnability would be important when we build a complex workflow. digdag is a workflow engine which syntax is simple and is able to run tasks with SQL, Python, Ruby, shell script, etc. digdag has Docker executor and it works like a charm with
How to ensure a new Docker image runnable with existing digdag workflow? I’ll show the way to run through it on CircleCI.
You can see the example repo on GitHub:
An issue with digdag Docker executor on CircleCI
Although CircleCI docker executor is the primary choice for CircleCI 2.0, which easily run with arbitrary Docker image, it doesn’t provide volume mount for docker since it launches remote sibling docker container. Hence digdag Docker executor assumes to mount a volume, like
-v /tmp:/tmp, you need some workaround to avoid it.
FileNotFoundError occurs in python operator in docker · Issue #649 · treasure-data/digdag
_HI. I am running the digdag server in the docker container with the following version. # docker –version Docker…_github.com(https://github.com/treasure-data/digdag/issues/649)
In this article, I’ll show you how to execute local mode digdag, a.k.a.
didgag run, on CircleCI with digdag docker executor.
Use CircleCI machine executor
tl;dr, use CircleCI machine executor, which runs VM on CircleCI.
version: 2jobs: test: working_directory: ~/app machine: image: ubuntu-1604:201903-01 docker_layer_caching: true steps: - checkout - run: name: Install digdag command: | curl -o ~/bin/digdag --create-dirs -L "https://dl.digdag.io/digdag-latest" chmod +x ~/bin/digdag echo 'export PATH="$HOME/bin:$PATH"' >> ~/.bashrc - run: name: Test digdag run command: | set -x digdag run test.dig
Machine executor has Python, Ruby, Java, and Docker CE by default, so you can easily run digdag on CircleCI.
Here are the dig file and Python script.
# test.dig+task: py>: test.show docker: image: "python:3.7-slim-buster"
# test.pydef show(): print("Hello CircleCI")
Build custom Docker image and test with digdag
In some cases, you want to test whether a new Docker image works appropriately with existing workflow.
If you build a new Docker image for digdag Docker executor and test with existing workflow, you can write like the following:
version: 2jobs: build_and_test: working_directory: ~/app machine: image: ubuntu-1604:201903-01 docker_layer_caching: true steps: - checkout - run: name: Install digdag command: | curl -o ~/bin/digdag --create-dirs -L "https://dl.digdag.io/digdag-latest" chmod +x ~/bin/digdag echo 'export PATH="$HOME/bin:$PATH"' >> ~/.bashrc - run: name: Build application Docker image command: | docker build -f ./Dockerfile -t chezou/my-image:latest . - run: name: Test treasure-boxes workflows command: | set -x digdag run test_custom.dig
Building a Docker image on CircleCI, you can use it form
digdag run command with the following workflow and Dockerfile.
# test_custom.dig+task: py>: test.show docker: image: "chezou/my-image:latest"
# DockerfileFROM python:3.7-slim-buster
RUN pip install tabula-py
- Using CircleCI’s machine executor enables to use
digdag runwith digdag Docker executor.
- It empowers us to do run through test for new Docker image with existing workflow on CircleCI
You can try it with this GitHub repo: