This is what I learned from creating a Python CLI (digdaglog2sql) in a day. In just 4 steps, you can release a CLI written in Python easily. Create a project by using poetry Poetry is a modern Python packaging and dependency management tool. Poetry is becoming popular and defacto rapidly. By using Poetry, it enables us to manage package dependency, to create a project template, and to publish to PyPI.

What’s data lineage? Data lineage is something to describe “Where this data comes from and where it goes?” I learned this term in my previous job. They provided “Cloudera Navigator” which includes data lineage from execution logs of Hive/Spark etc. lineage of Cloudera Navigator via https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cn_lineage_generation.html sqllineage is awesome open source tool for visualizing lineage Recently, I learned there is a Python package so called sqllinage, that makes analyze and

Hugo has a feature to show keyword based related articles. Yeah, keyword based articles might be useful, for people who can manage keyword, category, etc, constantly. I’d love to add content based recommendation that doesn’t require to write explicit keywords by myself. Then, I found an open source named “Prelims” which is developed by takuti. Prelims is a post-processing tool for Front matter of Hugo/Jekyll, that is a metadata of an article.

This article show how to develop a digdag Python workflow task efficiently.

Photo by Hitesh Choudhary on Unsplash Recently, I changed my CI from Travis to GitHub Actions. GitHub Actions is handy and useful for testing, publishing Python packages. Testing Python code on GitHub Actions Migration from Travis is super easy, just writing a simple workflow like: https://github.com/chezou/tabula-py/blob/master/.github/workflows/pythontest.yml The benefits of GitHub Actions for Python are: We can use build matrix (e.g., OS and Python versions) like Travis Launch time of GitHub is faster than Travis Easy for additional dependency installation by using uses syntax, which uses another workflow For example, installing JDK can be written as:

MLSE の本番適用のためのインフラと運用WGの活動を終了しました。

データリネージとは、DBなどでどのデータがどこから来てどこに行った、という「来歴」とも呼ばれる情報です。 自分がこの言葉を知ったのは、前職でC

prelimsとGitHub Actionsを使うことで、Hugoの自分の記事にレコメンド記事を表示できるようになります

去年に引き続き、年末の振り返りです。まだ西海岸時間は2021年です。 なにより、家族でバンクーバーに移住したのが大きな変化でした。 仕事 去年とチ

この記事はpyspa Advent Calender 2021の18日目です。昨日は@golden_luckyのリンゴの話でした。 今年も早いもので12月になってしまいまし

This article covers very biginning guide for MLOps, i.e., What is MLOps? How do tech giants make Machine Learning systems? What …

First book for how to design Machine Learning systems and how to proceed Machine Learning projects. This book is originally written in …